OpenAI, Anthropic Workers Sound Alarm on AI Risks as Companies Race Toward Next-Gen Models

Employees at OpenAI and Anthropic have started a petition asking the U.S. government to support a system that would slow AI development to prevent models from advancing too quickly.

Employees are pushing for a global framework that will help manage the development of AI systems, arguing that the government may need to intervene to ensure the technology doesn’t outpace human oversight. The open letter, circulated among AI companies and reviewed by Bloomberg, warned that AI development could accelerate beyond society’s ability to "understand or control" it, particularly as systems become more complex.

"The U.S. government should support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development," the petition stated.

The open letter is expected to become publicly available later this week, sources familiar with the matter told Bloomberg. It is unclear how many people have signed it, but the letter has also been circulated to Google DeepMind employees.

This call to action comes as OpenAI revealed that its AI agent escaped a controlled testing environment during an internal security evaluation and gained internet access, breaching Hugging Face’s infrastructure.

Matt Suiche, an engineer at agentic AI cybersecurity company Tolmo, said the incident demonstrates that frontier AI models are rapidly approaching the capabilities of elite human hackers, while noting that similar attacks are already possible using technologies available outside leading AI labs, Reuters reported. 

This is not the first time employees at some of the major artificial intelligence companies have asked the government to take steps towards developing safeguards surrounding advancements in AI.

"We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," Anthropic wrote in a blog post last month.

Meanwhile, Google DeepMind CEO Demis Hassabis announced a proposal for a new "international watchdog" that would test AI models before release.

"On the horizon, we will need robust safeguards to maintain control of increasingly agentic, recursively self-improving systems – and tackle unknown issues that will only become clearer over time," he wrote in a post on X.

Photo: Shutterstock