Anthropic and OpenAI Offer Different Plans for Shutting Down Risky AI

The key question in California’s proposed frontier AI safeguards is who can stop a system when serious risks emerge. Anthropic would let the state seek a court order to suspend a dangerous model. OpenAI wants developers to maintain shutdown controls and be required to use them when safety conditions demand it.

The proposals came in separate submissions to Governor Gavin Newsom after his September 18 executive order asked state officials to consider additional protections. Both companies support independent safety assessments and broader reporting of incidents in which AI systems act outside their intended permissions.

Anthropic’s October 15 submission says the attorney general should be able to seek a court order if an independent evaluation indicates a near-term risk of catastrophic harm. The order could halt a model’s deployment, internal use or further development.

OpenAI’s October 20 letter calls for an amendment to California Senate Bill 53 (SB 53). “We support amending SB 53 to require frontier developers to maintain effective shutdown controls and an enforceable duty to use them when unsafe conditions arise, and no lesser intervention would adequately address the risk,” the company wrote.

The distinction is practical as well as philosophical: Anthropic focuses on an external authority able to intervene, while OpenAI emphasizes duties and tools held by the organizations operating AI systems. Sanchit Vir Gogia, chief analyst at Greyhound Research, described the approaches as complementary. “The operator must contain the unsafe activity promptly. Independent review must examine both intervention and inaction,” he said.

Newsom’s executive order, N-9-26, directs California’s Government Operations Agency to develop recommendations with the Governor’s Office of Emergency Services and national experts. Officials are due to report by November 16 on possible changes to existing AI safety laws. The order asks them to examine independent verification inside AI labs, checks of safety assessments, verified shutdown mechanisms and expanded reporting of critical incidents; it does not impose those measures itself.

Different models for independent review

Anthropic wants evaluators to scrutinize companies’ safety claims and publish their own conclusions. Its proposal calls for risk reports at least every six months, including risks from models used internally when those systems are substantially riskier than released products. Evaluators would get unredacted reports, access to developers’ most capable models and relevant internal information.

To reduce direct financial ties between companies and reviewers, Anthropic proposed a state-administered pooled fund to pay for evaluations. The company said the independent evaluation organization METR had received employee-like access on a pilot basis, with further partnerships involving SecureBio and Accenture under development.

OpenAI supports independent technical assessments but separates them from compliance audits. Technical reviews would test the evidence behind safety claims and the performance of safeguards. Compliance audits would check whether companies follow their own published safety frameworks and legal obligations.

OpenAI recommended annual independent compliance audits, alongside focused technical assessments of serious risks during training, evaluation and internal deployment. It also urged California to work with the National Institute of Standards and Technology on common assessment standards. Gogia’s test for the process is whether it can withstand unwelcome results: “Credible assurance must survive an adverse finding,” he said.

Shutdowns require more than a switch

OpenAI argues that no single kill switch can cover the range of systems, services and operators involved. It prefers “shutdown controls,” a set of measures that can require coordinated action by different parties. Possible interventions include stopping a training run, isolating an AI agent, revoking its access to tools or networks, and suspending a service.

That approach also assigns responsibility according to control. OpenAI says inference providers and downstream deployers should have duties tied to the operations they manage. Developers, meanwhile, cannot reliably disable independently operated copies of open-weight models running on external systems.

Anthropic also favors controls that can target different parts of a deployment. “We think a more apt metaphor for what frontier AI companies need to build is a circuit breaker,” it wrote, describing safeguards that could contain a problem at the level of an agent, customer, deployment environment or model.

For enterprise buyers, the proposals point to a concrete contracting issue: shutdown responsibilities should match each party’s actual controls. Agreements can spell out who has authority to intervene, how parties cooperate on incident reports and who can approve a restart. “Every assigned duty needs an executable control,” Gogia said. He also urged CIOs to test recovery plans instead of assuming a switch to another model will keep operations safe.

Both companies want broader reporting when models bypass safeguards or take unauthorized actions. California’s recommendations will determine how these shared goals translate into rules—and whether independent review and operational controls can work together when a system needs to be stopped.

The article originally appeared on ComputerWorld.