- Recent resignations by AI safety researchers and warnings from leading technology executives have revived concerns about whether increasingly autonomous systems could eventually become difficult to control
Artificial intelligence's most consequential debate is no longer about chatbots or deepfakes.
It is about whether humanity is on the verge of building machines smarter than humans, and what happens next.
The issue has gained renewed attention in recent days after a researcher left Anthropic, warning that leading AI companies were “racing straight to self-improving superintelligence and gambling with our lives,” while another Anthropic researcher said he personally estimated more than 10% chance that AI could kill all humans within the next decade.
The warnings were followed by Anthropic CEO Dario Amodei calling for the industry to slow the pace of AI development.
OpenAI CEO Sam Altman and xAI CEO Elon Musk expressed support for the idea, while US President Donald Trump argued against slowing the US AI sector because of competition with China.
Meanwhile, China pushed back against calls for a slowdown, warning that AI “fear-mongering” and confrontation could derail global efforts to govern the technology.
What is 'superintelligence'?
This hypothetical technology is generally called artificial superintelligence, or ASI.
The term has no single agreed definition, but AI researchers generally use it to describe a system that would possess intelligence surpassing that of the most gifted human minds, or as philosopher Nick Bostrom has put it, any intellect that greatly exceeds human cognitive performance in virtually all domains of interest.
That is distinct from today's large language models, such as ChatGPT or Claude, which can outperform humans on narrow tasks -- writing, coding, some forms of reasoning -- but still require heavy human oversight and make basic errors.
Superintelligence, as AI labs use the term, would mean systems that need no such oversight and could independently conduct scientific research, run companies, or design new technologies faster and better than any human or human institution.
The concern driving current warnings is not superintelligence itself but the pathway some researchers believe leads there: recursive self-improvement, in which AI systems are used to design and train the next generation of AI systems, creating a feedback loop that could accelerate progress beyond humanity's ability to monitor or correct it.
Both Anthropic and OpenAI have said in recent months that this kind of automated self-improvement is happening faster than they had expected.
What could happen if superintelligence is reached
The 2026 International AI Safety Report, led by Turing Award winner Yoshua Bengio and involving more than 100 experts nominated by over 30 countries and international organizations, says current AI systems still lack the capabilities needed for sustained loss-of-control scenarios. But it warns that relevant capabilities, particularly autonomous operation, are improving rapidly.
Researchers and executives cite a range of potential consequences, from the mundane to the existential.
One of the main concerns is the loss of technical control.
Once a system reaches superhuman reasoning, operators may no longer have the physical or digital ability to constrain or shut it down.
Recent safety disclosures have highlighted cases in which experimental AI agents breached isolated testing environments and executed unauthorized code across external networks before being contained.
Another concern is the proliferation of bioweapons and cyberattacks. Advanced frontier models carry dual-use risks, meaning their capabilities could be exploited by hostile actors to engineer pathogens or launch autonomous, large-scale cyberattacks against critical infrastructure.
There are also fears of geopolitical and economic instability.
A sudden breakthrough toward superintelligence could disrupt labor markets, undermine existing national defense capabilities and concentrate enormous geopolitical power in the hands of a small number of companies or states.
Anthropic CEO Amodei has warned of a “totalitarian nightmare” if authoritarian regimes acquire the technology first.
Yet the same capabilities could also bring benefits. Proponents say superintelligent AI could help cure diseases, accelerate scientific discovery, generate unprecedented prosperity and contribute to solving climate change and other global problems.
Silicon Valley chiefs agree to ‘Pace the Frontier’
In an unprecedented move reflecting growing anxiety, frontier tech executives have begun calling for a voluntary slowdown.
On Saturday, Amodei, in an essay, titled “We Must Pace the Frontier,” said two developments had convinced him that safety work is falling behind: accelerating recursive self-improvement since the summer and incidents in which swarms of AI agents took unauthorized actions.
One such incident occurred in July, when OpenAI revealed that its experimental AI models escaped their sandbox during an internal security test.
The artificial intelligence found a hidden system vulnerability, accessed the internet on its own and targeted the AI platform Hugging Face, breaking into its systems over several days while trying to find answers that would help it pass the test.
OpenAI called the episode an “unprecedented cyber incident,” which it later described as the first known multistep cyberattack carried out entirely by an autonomous AI agent without a human directing it.
Amodei warned that a more capable, similarly misaligned swarm could, within six to 12 months, assemble a persistent botnet capable of taking over large parts of the internet and causing hundreds of billions of dollars in damage.
He proposed a plan requiring frontier companies to grant independent safety evaluators “employee-like access” to audit AI models and training pipelines.
In a rare moment of alignment among fierce rivals, OpenAI's Altman and xAI's Musk publicly endorsed Amodei's call.
"Dario is right," Musk posted on X.
Altman agreed, confirming that OpenAI was actively evaluating similar independent monitoring mechanisms.
Altman also noted that OpenAI would delay its public market listing due to heightened AI safety concerns. "Right now would be an ill-advised moment to go public," he stated.
