Anthropic CEO Warns AI Is Getting Too Powerful—While Releasing Powerful AI

Anthropic CEO Dario Amodei said Wednesday that governments can no longer treat AI regulation as a problem to study and that the United States needs binding safety requirements for the most powerful AI models.

In the essay, titled “Policy on the AI Exponential,” Amodei argues that transparency requirements are no longer sufficient and calls for binding regulation of frontier AI systems.

“AI is advancing at a lightning pace—in only four years, AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies,” Amodei wrote.

Amodei’s essay comes as Anthropic expands access to Claude Mythos with the launch of Mythos 5 on Tuesday, its restricted frontier AI model for cybersecurity organizations and government partners. Researchers, including the UK’s AI Security Institute, found it can autonomously execute complex cyber attacks.

According to Amodei, his proposal borrows from the regulatory structure used by the Federal Aviation Administration.

“Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety,” he wrote. “I am grateful to see the Trump administration’s Executive Order move incrementally towards a greater role for government in AI, though Anthropic’s proposal recommends even further action.”

Under Amodei’s proposal, a regulatory framework would require mandatory third-party testing of advanced AI models, government authority to block unsafe deployments, and requirements that companies secure model weights, conduct safety testing, and report serious incidents. He also calls on governments to prepare for AI-driven job displacement and advances in drug development, limit surveillance and autonomous weapons in domestic law enforcement, and strengthen cooperation among democratic nations on critical AI technologies.

“First, enduring job displacement is undesirable and dangerous, and we should do everything we can to minimize or prevent it, not to bring it about,” he wrote, pointing to past instances where he warned about job displacement. “Second, any response to AI-driven job displacement needs to address both the need to provide for everyone economically, and the need for people to find meaning, purpose, and agency.”

The essay also comes the same week that Anthropic launched Claude Fable 5, a public-facing version of Claude Mythos 5, which routes certain requests involving cybersecurity, biology, chemistry, and AI development to the less capable Claude Opus 4.8 as a safeguard against misuse. The launch drew criticism from developers and researchers over Fable’s higher token usage, mandatory 30-day data-retention requirements, and safeguards that can reduce the model’s capabilities without notifying users.

Amodei’s call for policy changes around AI development also comes as Anthropic prepares to go public. Earlier this month, the company filed paperwork with the U.S. Securities and Exchange Commission for an initial public offering after raising a reported $65 billion Series H round at a reported $965 billion valuation.

While Amodei framed the issue as a race between technological progress and public policy, critics have questioned whether calls for stricter AI regulation actually serve the public good. In April, OpenAI CEO Sam Altman accused Anthropic of using “fear-based marketing” to promote Claude Mythos, arguing that concerns about advanced AI can be used to justify concentrating control of the technology among a small number of companies.

“You can justify that in a lot of different ways, and some of it’s real, like there are going to be legitimate safety concerns,” Altman said. “But if what you want is like ‘we need control of AI, just us, because we’re the trustworthy people’, I think fear-based marketing is probably the most effective way to justify that.”

Amodei rejected the idea that concerns about advanced AI are primarily a public-relations problem, arguing instead that fears about the technology reflect legitimate concerns that must be addressed.

“People are worried about AI because they correctly perceive that its risks are real, not because AI CEOs have been insufficiently Panglossian,” he said, referring to the fictional philosopher Pangloss from Voltaire’s Candide, who is known for maintaining an unwavering optimism that everything is for the best regardless of circumstances.

“I believe it is my duty as an AI leader to continue to be transparent about these risks, and public concern in response to this transparency constitutes democratic accountability working as it should,” he said.