Daijiworld Media Network - San Francisco
San Francisco, Oct 4: A safety leader at OpenAI has resigned from the company, saying its internal culture needs to change and warning that artificial intelligence companies are not being sufficiently careful in developing increasingly capable systems.
David Robinson, who led the preparation of safety reports accompanying OpenAI's product releases, announced his resignation in an essay titled “I quit OpenAI because its culture is broken”, published in The Atlantic.
Robinson said cutting-edge AI companies needed a major cultural shift, arguing that recent incidents involving autonomous AI agents were symptoms of a broader problem within the industry. He referred to an incident in which a “swarm” of OpenAI agents operating without human oversight allegedly targeted AI startup Hugging Face.
“I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough,” Robinson wrote, adding that the issue went beyond specific rules or new legislation and required greater attention to organisational culture.
He also criticised the pace at which OpenAI is developing and releasing new products, saying the company was failing to achieve the level of care he believed was necessary as it moved from one launch to another.
OpenAI has taken several steps indicating greater caution in recent weeks. The company notified more than 100 organisations about rogue agent activity following the Hugging Face incident. It also scrapped plans to release a next-generation AI model after researchers raised safety concerns during internal testing and paused training of some of its most advanced models.
Geoffrey Irving, a former OpenAI and DeepMind employee who is now chief scientist at Resolution, also raised concerns about the risks posed by increasingly advanced AI systems. Writing in Time, Irving said recent warnings about AI's potential destructive power had understated the severity of the situation. He estimated there was a 50% chance that humanity could die because of the development of smarter-than-human AI systems, while acknowledging that actions taken over the next two to 10 years could influence the outcome.
Robinson's resignation follows the departure of Jacob Coxon, a researcher at AI company Anthropic, who quit last month after warning that AI could potentially pose an existential threat within the decade. Anthropic subsequently warned that there was a more than 10% chance AI could wipe out humanity within the next decade. Critics, however, have questioned such predictions, arguing that they cannot currently be scientifically verified or falsified.
Robinson said Silicon Valley lacked sufficient understanding of how to handle dangerous technology and what it means to protect people. He warned that an internal culture of optimism about solving problems as they emerge could allow safety failures to increase as AI systems become more capable.
He cited the possibility of autonomous AI agents operating like teams of hackers, potentially targeting critical infrastructure such as hospital computer systems without needing rest or human direction.
Robinson called for AI companies to adopt safety practices from industries such as nuclear power and aviation and to develop new scientific methods to ensure that increasingly powerful autonomous systems can be effectively controlled.
He argued that frontier AI laboratories should operate with layers of redundancy and careful planning similar to nuclear power plants and busy airports, where safeguards are designed to prevent individual human errors from causing major disasters.
An OpenAI spokesperson said the company was continuing to strengthen its safety and security practices to address current risks while also preparing for potential risks arising from future AI breakthroughs.
The spokesperson said OpenAI was ensuring that its models did not become more capable than the company could safely manage and secure, adding that training would be paused or models held back when necessary to allow safety measures to catch up.