An OpenAI security leader has resigned from the company, warning that its culture was broken and that AI companies were not “being careful enough” when developing the technology. David Robinson, who led the writing of the security reports that accompanied the ChatGPT developer’s product releases, explained his resignation in an essay titled: “I left OpenAI because its culture is broken.” autonomously and without human supervision, attacking AI startup Hugging Face was “typical of the industry, given the speed and flexibility with which people operate.” In an article in Atlantic magazine, Robinson wrote: “I agree with other recently deceased staff members that the companies building this technology are not being careful enough. But I think we need to look beyond specific rules or new laws. We need to talk about culture.” the next, it fails to reach the level of care that I think is needed.” However, OpenAI has shown signs of caution in recent weeks following the Hugging Face incident and the revelation that it has notified more than 100 organizations about rogue agent activity. This week it announced it would scrap the release of a next-generation AI model after researchers raised security concerns during internal testing. OpenAI has also paused training its most advanced models. Geoffrey Irving, who worked on OpenAI, and was chief scientist at the UK government’s AI Safety Institute before joining AI safety research firm Resolution, also issued a warning about AI on Saturday Writing in Time, he said: “Recent warnings about the potential destructive power of AI are underestimating the seriousness of the situation. I think there is about a 50% chance that we will all die due to the development of smarter than human AI. systems, and that our actions over the next two to 10 years will determine the outcome.” Robinson’s essay also follows the resignation of Jacob Coxon, a researcher at OpenAI rival Anthropic, who resigned from chatbot developer Claude last month. He warned that AI “could kill us all before the end of the decade,” and was followed by a warning from Anthropic that there was more than a 10% chance that AI would wipe out humanity in the next decade. Critics of such warnings They have warned, however, that they are not scientific because they cannot be verified or falsified. Robinson wrote that Silicon Valley lacked awareness of “how to handle dangerous technology” and “what it means to take care of people.” Warning that OpenAI had “unbounded optimism” about solving problems as they arose, he wrote that this internal culture meant that security failures would only increase as systems became more capable. “Imagine ‘rogue’ agents working as teams of hackers. (for example, holding hospital computer systems for ransom) but never need to sleep,” Robinson wrote. Relying on security expertise in other fields such as nuclear and aviation and developing “new science” that ensures powerful systems of the future are capable of being controlled when operating autonomously. “Given today’s risks, frontier laboratories must function like nuclear power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that occasional and inevitable human error does not open a door to disaster,” he wrote. An OpenAI spokesperson said the company was continuing to “strengthen our security and security practices to address the risks we see today,” as we work to address risks that could be created by future AI advances. “We are ensuring that our models do not become more capable than we can safely manage and protect, and we pause training or hold models when we need to slow down,” the spokesperson said.
