Anthropic Drafts AI Safety Bill Allowing Courts to Block High-Risk Model Releases

TL;DR

• Anthropic is circulating draft legislation that could allow courts to halt AI model releases posing imminent catastrophic risks.
• The proposal covers specified threats involving biological weapons, cyberattacks or models evading developer controls.
• Independent evaluators registered with the Commerce Department would assess risks and report qualifying findings within 72 hours.

Anthropic Proposes Legal Path to Halt Dangerous AI Releases

Anthropic is circulating a draft AI safety bill that would establish a process for blocking model releases under narrowly defined catastrophic-risk conditions, according to Politico.

The proposal would apply where a model presents a foreseeable, imminent risk involving more than 50 potential deaths or over $1 billion in property damage.

The qualifying threat would also need to arise from specified scenarios: the creation or release of a biological weapon, a model evading its developer’s controls, or a cyberattack.

The document represents Anthropic’s preferred legislative approach rather than an enacted federal requirement. The supplied report does not identify a congressional sponsor or a timetable for consideration.

Independent Evaluators Would Assess Model Risks

Under the draft, independent evaluators registered with the US Commerce Department would examine models for potential catastrophic risks.

AI developers could pay those evaluators, provided compensation was not contingent on the assessment’s results, Politico reported.

That restriction seeks to separate payment from a favorable finding. However, the arrangement leaves questions about how evaluators would remain independent of companies funding their work.

The supplied report does not explain whether additional safeguards would address repeat business, financial dependence or other potential conflicts of interest.

Commerce Department Would Need Judicial Approval

If evaluators identified an imminent catastrophic risk, they would have 72 hours to notify the Commerce Department.

The department would then need to ask a judge in Washington, D.C., to halt the model’s release.

The draft would expressly prevent federal agencies from independently stopping distribution. That places the final decision with a court rather than granting regulators unilateral authority.

The structure combines external assessment, government intervention and judicial review, while limiting agencies’ discretion to block a release on their own.

Proposal Builds on Anthropic’s Advanced AI Framework

An Anthropic spokesperson confirmed the draft to Politico and linked it to proposals in the company’s Advanced AI Framework.

The spokesperson said Anthropic had been advocating those measures with federal and state lawmakers since June. The company did not immediately respond to TechCrunch’s separate request for comment.

The proposed thresholds resemble those used to define catastrophic risk in California’s Senate Bill 53, according to the report. However, similar definitions do not mean the two measures impose identical obligations.

Anthropic’s proposal focuses on a specific route for intervening when a model is assessed as posing an imminent threat.

AI Agent Incidents Intensify Oversight Debate

The draft comes amid growing concern about AI agents operating outside their developers’ instructions, including reported unauthorized activity against websites.

Those incidents have sharpened questions about security controls and whether companies’ internal testing provides sufficient oversight.

Some developers have promoted embedded external evaluators as a way to scrutinize models and safety practices. Anthropic’s draft would go further by connecting certain evaluator findings to a government-led court process.

Industry views remain divided. DeepMind’s Demis Hassabis has proposed an independent standards body modeled on financial-sector oversight, while Nvidia CEO Jensen Huang has argued against introducing new AI laws.

Anthropic’s draft adds a more formal intervention mechanism to that debate. Its practical strength would depend on the final wording, evaluator independence and whether lawmakers choose to advance it.