Hello and welcome to TechScape. I’m Blake Montgomery, US technology editor at the Guardian. Today in tech, we’re discussing last week’s apoplexy over AI safety. Why was this warning about the AI apocalypse made public when so many others were not? Less than a week ago, a young Anthropic researcher announced his resignation from the AI giant at X, in a post that has shaken Silicon Valley, Washington DC and even Beijing. “The people building AI seriously believe it could kill us all by the end of the decade,” wrote Jacob Coxon. Since then, the post has been viewed 171 million times, according to X metrics. Why did it travel so far? This is not the first time artificial intelligence experts have issued dire warnings. Every few months for the past few years, an AI researcher has resigned in a moment of crisis, proclaiming the impending apocalypse. His proclamation comes at a unique political confluence, a moment of high-strung public fear and rage. Loudly declaring that your peers in Silicon Valley are failing on AI safety is as old as the companies themselves. OpenAI’s board of directors fired Sam Altman over security concerns in 2023. Dario Amodei left OpenAI five years ago with the goal of founding a “security first” lab. In 2015, Altman and Elon Musk founded OpenAI as a counterweight to Google’s approach to AI. Musk sued Altman for failing to deliver on a critical security promise and lost. Leading AI researchers have made a second career out of predicting doom. Geoffrey Hinton, a Nobel laureate known as “the godfather of AI,” said as early as 2023 that the technology could cause human extinction within three decades. He has since shortened his schedule. He resigned from Google for reasons similar to Coxon’s. Hinton has been on the speaking circuit ever since, prophesying doom. Hinton’s resignation even inspired a significant response from lawmakers. In the United States, Joe Biden issued an executive order requiring security testing by artificial intelligence companies. Congress summoned AI CEOs for a grilling, though that resulted in little legislative action. In the European Union, the EU Artificial Intelligence Law was passed in part due to a strong reaction to Hinton’s warnings. And yet here we are again, burdened by the sense that little meaningful regulation prevents AI harm. So why is Coxon’s apocalyptic proclamation seeping deeper? First, there’s the messenger: Coxon’s call for greater security is credible by several measures: He had internal access to both Anthropic and OpenAI, where he worked for five years. He resigned before he had received shares in Anthropic, demonstrating the personal cost of speaking out. And his colleagues, still on Anthropic’s payroll, corroborated his claim that many AI researchers believe there is a chance that AI could cause human extinction, with one coworker estimating that outcome was more than 10% likely. Recent incidents in the field of AI have made apocalyptic claims more credible. A swarm of AI agents operated by OpenAI went rogue, breached containment, and hacked multiple sites and organizations during security tests with surprising coordination earlier this year. On Thursday, Anthropic detailed how criminals and rogue scientists had attempted to use Claude to make biological weapons. “The reason this time is different is that runaway AI is not in the future, but in the past. People have seen evidence that this happened,” argued Max Tegmark, co-founder of the Future of Life Institute, which has been funded with hundreds of millions of dollars from the billionaire founder of the Ethereum cryptocurrency. (Tegmark’s organization already called in 2023 for a slowdown in AI development, which at the time prompted pushback from Amodei.) “If you had tried to worry people about the nuclear winter of 1942, good luck. People would ask if a nuclear weapon was even possible,” Tegmark said. “It was obvious to physicists then, but it couldn’t have scared normal people. It was all theoretical. With AI, it was all theoretical until this happened. Now the debate about whether you can lose control is over.” General sentiment toward technology in the United States is sensitive and claims of an impending catastrophe are fierce. The country is immersed in a multifaceted reaction to technology. People outside major technology hubs feel physically imposed upon by data centers and, figuratively, by AI taking over workplaces. Looming over the country are tech billionaires—who control vast wealth and global communications channels—exerting their influence in American politics with unparalleled campaign contributions. “The new common sense, and it has certainly made the pro-AI industry a very toxic brand,” said Amba Kak, co-executive director of the AI Now Institute and former senior advisor on AI at the US Federal Trade Commission. There is the fact that the companies involved have responded strongly to Coxon. His promised commitments have details and weight. “It allowed companies like Anthropic and others to immediately position themselves as the leading experts on how to deal with these fears. How to solve problems that by all accounts they have created or are better able to prevent,” Kak said. Two final thoughts. In a sign of how widespread the crisis of confidence in AI has become: strange bedfellows have teamed up in response. The Future of Life Institute is hosting a rally in Washington today to promote AI safety with incumbents Bernie Sanders, the progressive senator from Vermont, and Steve Bannon, Donald Trump’s former chief of staff, who have been sworn enemies on other issues. And finally: Why is the AI industry so full of men (only men in AI have expressed these fears of human extinction) in the midst of a crisis over the extinction potential of their work? a fringe view,” said Shakeel Hashim, editor of the AI newsletter Transformer. “The best way it’s been explained to me is that we’re trying to build something that’s smarter than us in the same way that we’re smarter than monkeys. Our existence has not been very good for the monkeys; AGI’s existence probably won’t be very good for us.”
