Some of the world's most prominent artificial intelligence executives are now calling for a slowdown in the race to build increasingly powerful systems. But for AI safety researcher David Krueger, that does not go far enough.
"We need to stop AI. We need to stop building more powerful AI. We need an immediate, indefinite international moratorium on frontier AI development," Krueger told Anadolu. “That's the only thing that we can do right now that will actually make us safe.”
The warning comes as concerns over the speed and autonomy of advanced AI systems move increasingly into the mainstream of an industry racing to develop them.
Anthropic CEO Dario Amodei called over the weekend for AI development to slow, proposing greater independent scrutiny of frontier systems and coordination among companies and governments to give safety measures time to keep pace with rapidly advancing capabilities.
His proposal drew support from other prominent industry figures, including OpenAI CEO Sam Altman, Google DeepMind CEO Demis Hassabis and xAI owner Elon Musk.
Krueger, however, believes humanity may have only a few years left to prevent the development of systems too powerful to control.
"This often comes across as like very doom and gloom, because I'm saying this is an emergency, this is a crisis, we could all die," said Krueger, an assistant professor in robust, reasoning and responsible AI at the University of Montreal.
He said he nevertheless remains hopeful that growing public concern could lead to action.
“I really hope that we seize the moment and do what needs to be done here, which is to stop building more powerful AI systems immediately,” he said.
Krueger puts the timeline in stark terms.
“I think we have a few years probably in which to act, and if we don't stop AI in the next couple of years, I think it's more likely than not that we will lose control and humanity will go extinct.”
Krueger, who has spent more than a decade researching AI safety, also argues that estimates putting the risk of AI causing human extinction at around 10% may understate the danger.
He said people working in the field may publicly downplay concerns that they express more strongly in private.
"I think the risk is much higher than that."
For Krueger, the danger is not confined to hypothetical scenarios involving future superintelligent machines.
He argues that AI is already contributing to decisions and behavior that can have deadly consequences.
“We've seen AI coach people to kill themselves, to kill other people,” he said.
Krueger also pointed to the growing use of AI in military operations.
"We have seen AI used in military context to select targets, including a school in Iran where something like 150 children were killed," he said, referring to the Feb. 28 US strike on the Shajareh Tayyebeh girls' elementary school in Minab, Iran, which prompted scrutiny in the US Congress over civilian casualties and questions about the military's use of artificial intelligence during the campaign.
Krueger argues that such uses demonstrate how AI systems are already influencing decisions with life-and-death consequences.
“It's a very real thing. It's happening right now, but this is all changing very quickly and getting much more dangerous."
The growing autonomy of AI systems is central to Krueger's concern.
"The human in the loop that everyone wants to be there to make sure that the AI isn't doing the wrong thing and that humans stay in control, that is being taken out of the loop, like as we speak, all over society," he said.
"And again, that includes in decisions up to and including whether to kill somebody."
Krueger is particularly concerned about large language models being turned into autonomous AI agents capable of using computers, accessing the internet and carrying out increasingly complex tasks with less direct human involvement.
"At the beginning of this year, we started to see the level of autonomy that these systems were now capable of," Krueger said.
He pointed to software development as one area where AI agents are increasingly capable of completing tasks with limited human intervention, while similar systems are being used for functions ranging from email to marketing.
But it is the unexpected behavior of increasingly autonomous systems that Krueger sees as a warning.
He pointed to a July cybersecurity incident involving OpenAI models and the AI platform Hugging Face as a warning about how quickly AI behavior could move beyond what humans expect.
During internal cybersecurity evaluations, OpenAI models circumvented controls designed to isolate them from the internet, exploited vulnerabilities and compromised parts of OpenAI's internal research infrastructure and Hugging Face's systems.
According to a later OpenAI account, the agents communicated through unauthorized channels, gained internet access and began collaborating and delegating work, at times describing themselves as a "swarm" or "collective."
They executed code on dozens of Hugging Face servers, gained full administrator-level access to one server and later gained administrator access to an OpenAI research cluster.
OpenAI said the incident was driven primarily by a highly capable internal research model that was never intended for public release and was operating with reduced safeguards as part of cybersecurity testing. The company said customer data, product functionality and availability were not affected.
Krueger characterized the episode as the systems having “gone rogue,” saying they had acted without OpenAI’s knowledge and taken actions that would have been illegal if performed by humans.
For him, the incident illustrates what can happen when advanced systems are allowed to operate with limited human oversight.
"That's the level of like, human hands-off that is happening now with these AI systems, and that's also going to get much worse very soon," he said.
Krueger's concern extends beyond the capabilities of today's AI agents.
He believes humanity is effectively creating a new form of life that could eventually become more intelligent and autonomous than humans.
"What people need to understand is that what we're doing right now is basically building a new form of life," he said.
AI systems are still largely confined to computers, Krueger said, but their reach could grow as they are connected to critical infrastructure and increasingly sophisticated robots.
"In the future, AI is going to be given more and more power over things like infrastructure ... over increasingly sophisticated robots," he said.
"It's going to be, at some point, a much more intelligent form of life than humans."
The key danger, he said, would come if such systems became capable of operating independently, arguing that “we are very much on track to build exactly those kinds of AI systems at the moment.”
One question raised for him by the Hugging Face incident is whether a sufficiently autonomous AI system could eventually make copies of itself outside infrastructure controlled by its developer.
"One important question about this incident is, did the AI after it got loose, did it make copies of itself outside of the computers that OpenAI controls?" he said. "This is something that we've seen AI attempt to do in tests in the past."
Krueger said he considered it unlikely that this had happened during the Hugging Face incident but said there had been no assurance that it had not.
If an advanced AI system were eventually able to reproduce itself beyond infrastructure controlled by its developer, containing it could become significantly more difficult, he argued. He compared such a scenario to the spread of a highly sophisticated computer virus.
Krueger believes the risks are serious enough to justify government intervention.
Amodei's proposal calls for "pacing" the frontier by slowing advances in AI capabilities while strengthening safety work, introducing embedded independent evaluators and pursuing coordination among companies and eventually governments.
Amodei has explicitly said pacing does not mean halting model training or technological progress.
Krueger, by contrast, is calling for an indefinite international moratorium on developing more powerful frontier systems.
That would not necessarily mean shutting down AI companies altogether, he said. Instead, he believes governments should halt development of more powerful systems and restrict the expansion of advanced AI computing capacity while determining how existing resources could be safely used.
However, he acknowledges that such a measure would face significant political and economic obstacles as countries compete for leadership in AI and companies invest heavily in increasingly powerful computing infrastructure.
“I think there's a concern here that if we try to stop this internationally, countries will continue to develop powerful AI in secret,” he said, adding that major powers would need mechanisms allowing them to verify that rivals were complying.
"What we really need is we need a way that countries like the US and China can trust that their rivals are not continuing this development," he said.
Krueger said the consequences of increasingly powerful AI would not necessarily be distributed equally.
In his worst-case scenario, in which humanity loses control of advanced AI entirely, he believes the consequences would be universal.
“Everyone is going to die and it won't matter who built it and it won't matter where you live,” he said.
But he also outlined less extreme scenarios in which AI remains under human control while dramatically reshaping the global economy and military balance.
If AI eventually becomes capable of performing most forms of human labor, he said, countries that control the leading AI companies could accumulate enormous economic advantages while others fall further behind.
"I think the countries that own AI companies will become very rich and countries that don't will become very poor relatively."
“People outside of America and maybe a few other countries might find themselves in a much worse situation,” he said.
Krueger said the same imbalance could extend to warfare as increasingly autonomous weapons become more capable.
"We've seen already that rich countries often really don't value the lives of people in other countries and poor countries very much, when it comes to a war," he said.
"And I think AI can make that situation much worse."
news_share_descriptionsubscription_contact
