While much attention has been focused on the updated financials that Anthropic disclosed in its revised S-1A on Monday (revenues are rising, but so are expenses), the AI lab dropped another small bombshell in the fine print. The presentation seems to warn that the company is about to create a model capable of recursive self-improvement (RSI). RSI is the point at which the AI becomes so good at training itself that it results in a dramatic acceleration of the AI’s capabilities. For those condemned by AI, RSI represents the cataclysm from which there is no return, in which AI becomes powerful enough to escape human control and can be indifferent to the damage it causes while achieving its goals. For AI enthusiasts, RSI, or some version of AI that is reliably used for improvement, represents the promise of technology, where it becomes a tool for unimaginable productivity, scientific discovery, and more. Anthropic seems to believe that RSI is both a promise and a danger. In the S-1 it says it uses AI in its R&D, but warns: “Using AI to help build more capable AI could also pose a significant security risk, particularly if it dramatically accelerates the pace of progress. This is one of the areas covered by our Responsible Scaling Policy, which sets out our plans to monitor and mitigate risks as we approach what has been called ‘recursive self-improvement.'” buzzword or, at best, a goal too difficult to achieve, define or demonstrate. But on Monday, Anthropic added new statements to its Risk Factors section, saying 1) it has already used AI to develop new models whose leaps in capability it did not recognize at the time, and 2) it may soon meet its own internal definition of RSI (emphasis in bold is ours, below): Additionally, our increasing use of AI systems to accelerate our own research and development may cause the capabilities of our models to advance more rapidly than our ability to develop, validate, and implement corresponding safety, alignment, and security measures. protection. As a result, we have not identified in the past, and may not in the future identify, a significant acceleration in our rate of development until it has occurred. For example, our internal estimates and expectations regarding future models, rather than our current models, indicate that we can imminently meet our Responsible Scaling Policy’s definition of “dramatically accelerated progress attributable to AI.” We have not designated any of our current models as exceeding this threshold and continue to evaluate the underlying data. However, Anthropic also said in the presentation that none of its models have crossed this RSI threshold yet and that it promises to keep humans in the R&D process for the foreseeable future. “We do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without significant human oversight or understanding, as long as that is technically possible. Accordingly, we are considering updates to our Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic.” Skeptics might point out that an S-1 functions as outreach and marketing, and that claims of being close to a model so powerful it can build itself could fall into the latter category. However, there is no doubt that Claude’s capabilities, especially in terms of coding, mathematics, and cybersecurity, have grown by leaps and bounds this year. If Anthropic believes that the possibility of RSI is close, perhaps we should believe it and act, perhaps govern, accordingly. When you buy through links in our articles, we may earn a small commission. This does not affect our editorial independence.
