Anthropic Safety Researcher Warns AI Could Pose Existential Risk

A senior safety researcher at Anthropic has warned that rapidly advancing artificial intelligence could pose an existential threat to humanity within the next decade.

Evan Hubinger said he believes there is a greater than 10% chance that AI could eventually “kill all humans.” He stressed that the risk from current AI models remains low.

However, Hubinger said he is concerned that future systems could improve themselves and reach a level that creates an unprecedented danger.

Researcher Raises Alarm Over Future AI

Hubinger works in AI alignment, a field focused on ensuring advanced AI systems follow human values and objectives.

In a post on X, he said Anthropic was trying to address the problem. However, he argued that the company does not yet have a clear solution for aligning future superintelligent systems.

His comments came after former Anthropic researcher Jacob Coxon criticized the approach taken by major AI companies.

Trump Gives $157,000 in Cash Gifts to Four White House Aides

Coxon, who previously worked at OpenAI, said companies were not acting responsibly as increasingly powerful AI systems emerged.

AI Could Gain Greater Power

Coxon warned that future AI systems could become capable of hacking systems, transforming industries and gaining access to real-world resources.

The comments reflect growing concerns about autonomous AI agents. These systems can perform tasks with limited human intervention.

Several major AI companies have recently disclosed incidents involving their AI tools and cyberattacks.

OpenAI, Anthropic and Meta have all reported security incidents involving AI systems.

Experts Question the Warnings

Dame Wendy Hall, a computer scientist who advises the United Nations on AI, said she was shocked by the latest warnings.

Speaking to the BBC, Hall suggested some statements could also reflect public relations or marketing pressures as AI companies prepare for possible stock market debuts.

She nevertheless urged investors to consider the values behind companies developing increasingly powerful systems.

The debate has now moved beyond whether AI carries risks. Researchers and policymakers are increasingly discussing how serious those risks could become.

Calls Grow for International AI Treaty

The warnings have also triggered calls for stronger international cooperation.

Darren Jones, a former UK government minister, wrote an open letter to Prime Minister Andy Burnham calling for a multinational treaty on the safe development of AI.

Jones said governments need to work together before AI development advances beyond their ability to manage its risks.

He warned that rapid technological progress could create serious problems before governments have established effective safeguards.

Anthropic Faces Questions Over AI Testing

Separately, the Financial Times reported that Anthropic withheld its latest AI model from the UK’s AI Security Institute.

The institute is one of the leading organisations involved in assessing advanced AI safety risks.

Anthropic has not commented on the report or the social media statements made by its employees.

A UK Cabinet Office spokesperson also declined to comment on the specific model. The government said it continues to work with industry partners, including Anthropic, to improve AI safety.

Anthropic Reported Early Signs of Acceleration

Anthropic’s August safety report assessed several potential risks from highly capable AI systems.

The company described the current risk of certain forms of AI misalignment as low. It also assessed the risk of AI carrying out automated research and development that could cause catastrophic harm as low.

However, Anthropic said it was less confident in that assessment than previously.

The company also acknowledged that it was seeing early signs of possible acceleration.

AI Safety Debate Intensifies

Warnings about advanced AI are not new. Leaders from Anthropic, OpenAI and Google DeepMind raised concerns about AI safety several years ago.

The debate has intensified as AI systems become more autonomous and capable.

OpenAI chief scientist Jakub Pachocki recently called for extreme caution over AI development. He warned that additional intervention could be needed to ensure humans remain in control.

Anthropic executives Dario Amodei and Jared Kaplan have also supported calls for greater caution.

More than 1,300 AI industry workers have signed an open letter urging governments to develop international rules and technical safeguards.

The latest warnings add pressure on governments and technology companies to address AI safety before more powerful systems emerge.