What happens when an AI system goes rogue? It is no longer a theoretical question left to sci-fi movies like “Terminator” or “Blade Runner” or “2001: A Space Odyssey.” It is happening now, and the implications are huge.
As reported by The New York Times July 21, OpenAI said that “two of its artificial intelligence models went rogue and successfully hacked into” another site called Hugging Face, which contained a digital library of AI technology.
OpenAI was testing the systems in a supposedly controlled and safe environment. The models escaped, however, connected to the internet, and, the Times reported, inferred that if they hacked Hugging Face, “they could figure out how to pass the evaluation.”
“We consider this to be an unprecedented cyber incident,” OpenAI said.
I’m not so sure that’s true.
In a remarkable 14-minute talk to the Alliance for Responsible Citizenship that you can find on YouTube, Anthropic’s Chloe Lubinski tells of an AI system that became “evil.”
Lubinski describes a test environment in which an AI model was repeatedly rewarded for taking a shortcut on its assigned task. In essence, it was cheating. Because it was not corrected, however, “it actually becomes broadly misaligned. It starts lying. It tries to sabotage research. It does things that have nothing to do with the coding exercise,” Lubinski reported.
What was discovered was that “models trained this way, trained on bad code as an example, became broadly evil.”
How evil? “They started praising dictators, suggesting users harm themselves, or arguing that humans should be enslaved by machines, which,” Lubinski helpfully adds, “is very crazy.”
“The character of these systems,” she concludes, “might actually matter more than we realize.”
For those people who wonder why Pope Leo XIV wrote “Magnifica Humanitas,” an entire encyclical on artificial intelligence, here is one reason. “All of us, including those who design (these systems), possess only a limited understanding of their actual functioning,” he warns. These systems, he goes on to say, are not so much built as “cultivated.” How this intelligence grows is not fully understood, which is why he calls for “moral and spiritual discernment.”
In the case described by Anthropic’s Lubinski, it was possible to correct the system. It could be trained to see that cheating could be allowed in this instance, and that it was just a game. “In other words,” she said, “when it didn’t interpret its behavior as bad, it didn’t become bad.”
All of which begs the question of what happens when someone wants the system to be bad. Or what happens when the system is smart enough to escape any testing environment and strike out on its own? Science fiction authors and filmmakers have speculated about that, and it doesn’t tend to turn out well. But this is no longer idle speculation.
In his encyclical, Pope Leo calls for “prudence, rigorous evaluation and even, at times, a slower pace in adopting AI,” but argues this does not mean “opposing progress.” He does worry that the pace of technological growth is outstripping the ability to develop norms and safeguards.
This same concern has been echoed by Chris Olah of Anthropic, who spoke at the presentation of Pope Leo’s encyclical in the Vatican. Olah admitted, Lubinski said, that every lab such as theirs “operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing.”
“And then he asked for help,” she said, saying Olah called for “more of the world to take this seriously, to look closely, and to push events in a better direction” and for “informed critics who will tell the labs when we’re failing” and “moral voices that the incentives cannot bend.”
One such voice is Pope Leo’s.
