A Catholic expert on artificial intelligence is commending recent calls by AI research firm Anthropic to rein in the technology amid a significant acceleration in its growth.
Anthropic — which recently joined Pope Leo XIV in unveiling his new encyclical on AI — broadly called for “a meaningful slowdown or pause” in advanced AI development, as the technology is poised to accelerate “sooner than most institutions are prepared for.”
That message was issued in a June 4 post on the Anthropic Institute website, with authors Marina Favaro and Jack Clark explaining the concept of “recursive self-improvement” — or, in the words of their article’s title, “When AI builds itself.”
Through recursive self-improvement, an AI system uses existing AI technology to improve upon itself, potentially becoming “capable of fully autonomously designing and developing its successor,” Favaro and Clark said.
Recursive self-improvement is “effectively the creation of an automated AI engineer, or many, many, many automated AI engineers (as many as they have the computational capacity to run),” explained Brian Patrick Green, director of technology ethics at the Markkula Center for Applied Ethics at Santa Clara University. “Rather than humans programming AI, AI becomes powerful enough to program itself.”
Green added that such self-improvement is “one of the goals for AGI,” or “artificial general intelligence.”
But, warned Favaro and Clark, while self-building AI could bring either “enormous good for the world” — for example, in science and healthcare — “full recursive self-improvement also might increase the risks of humans losing control over AI systems.”
Others hold that the potential loss of human oversight is more nuanced, with humans retaining varying levels of control amid rapid AI growth, as author Mathew Hutson noted in a May 7 article for IEEE Spectrum, the flagship publication of the global technical professional organization IEEE, the Institute of Electrical and Electronics Engineers.
Still, said Green, Anthropic’s concerns are well-founded, especially given the critical role of alignment, or the process of ensuring the technology squares with human values, so that AI models safely serve human interests.
“Recursive self-improvement can lead to a situation called ‘values lock-in,'” said Green.
In such a scenario, he said, “whatever the values are at the start of improvement become locked-in for all subsequent models because the machine is really in charge of making new versions of itself.”
That includes “choosing its values,” Green said, noting that “humans might no longer be in a position to control the machine’s behavior.”
“It is very important to get recursive self-improvement right, otherwise it could spiral out of control and create new versions of itself that are not friendly,” he said.
Favaro and Clark underscored that any downshifting in frontier AI development would require a “global coordination mechanism” in order to prevent bad actors from exploiting such a pause.
Green also commended Anthropic’s move to comply (albeit grudgingly) with a June 12 Trump administration order to suspend foreign access to the company’s Fable 5 and Mythos 5 models due to national security concerns.
The two AI models have, among other features, strong performance levels in agentic coding (displaying reasoning and programming abilities), knowledge work (providing expertise and critical thinking) and spatial reasoning (representing objects in relation to each other).
In a statement that same day, Anthropic said it would remove access to the two AI models “for all users,” stressing, “We disagree that the finding of a narrow potential jailbreak should be cause for recalling a commercial model deployed to hundreds of millions of people.”
The company reiterated its earlier statement that “government should have the ability to block unsafe deployments, as part of a statutory process that is transparent, fair, clear, and grounded in technical facts.”
However, said Anthropic, “this action does not adhere to those principles.”
The company regarded the Trump directive as “a misunderstanding,” and planned to “restore access as soon as possible.”
Regarding the federal halt, Green said it was “better to be safe than sorry.”
“Mythos has demonstrated powerful cyberoffense capabilities, and it is understandable that the U.S. government would not want enemy nations to gain access to such a tool,” he said.
He noted that “Anthropic itself has recommended this caution, so the disagreement seems to be whether the restrictions are good enough, which is a matter of judgment.”
Green admitted that “in some ways, the restrictions might be too stringent in that even Anthropic’s own non-U.S. citizen employees — many of whom are from friendly nations like Canada, the U.K. and Germany — are no longer allowed access.”
That move signals “direct government interference with business activity, and should cause American businesses some concern about government overreach,” said Green.
