Hi Friends,

Even as I launch this today ( my 80th Birthday ), I realize that there is yet so much to say and do. There is just no time to look back, no time to wonder,"Will anyone read these pages?"

With regards,
Hemen Parekh
27 June 2013

Now as I approach my 90th birthday ( 27 June 2023 ) , I invite you to visit my Digital Avatar ( www.hemenparekh.ai ) – and continue chatting with me , even when I am no more here physically

Translate

Wednesday, 21 October 2026

Fifty Deaths or One Billion Dollars — and Then You Need a Judge's Permission

Fifty Deaths or One Billion Dollars — and Then You Need a Judge's Permission
TL;DR Anthropic has drafted a US federal AI safety bill that, for the first time in American statute if it passes, would put a legal gate in front of a model release. But the gate opens on a body count rather than a capability test, the inspectors may be paid by the inspected, and only a D.C. judge can turn the handle. Three years after I proposed an approval certificate for chatbots, my gate has come back wearing a corporate logo. It should be fixed before it becomes the template.

What happened

Politico's Owen Dahlkamp obtained a confidential draft bill that Anthropic has been circulating on Capitol Hill and published it on 16 October. Anthropic confirmed the draft. The text is dated 29 September and has been given working titles including the Artificial Intelligence Evaluation, Governance, and Incident Safety (AEGIS) Act, according to the public tracker at aipolicy.tech. TechCrunch's account landed late on 20 October US time, which is why the follow-up wave, much of it fixed on the judge-only clause, is being read here this morning.

  • The trigger. A "foreseeable, imminent risk" of more than 50 deaths or serious injuries, or more than $1 billion in property damage, arising from a biological weapon, the model evading its developer's controls, or a cyberattack (Politico, TechCrunch).
  • The inspectors. Independent evaluators registered with the US Commerce Department would assess frontier models periodically. Labs may pay them, provided the payment is not contingent on the result. An evaluator that finds an imminent risk must notify Commerce within 72 hours.
  • The switch. Commerce can then ask the US District Court for the District of Columbia to halt the model. The order runs for 45 days at a time. The draft expressly bars every other department, agency and officer from suspending a model's development, deployment or internal use.
  • The price. The bill would preempt some state AI laws on disclosure, third-party evaluation and incident reporting.

Anthropic says the text turns its Advanced AI Framework, promoted to lawmakers since June, into legislative language. Its federal affairs lead, Anthony Cimino, wrote on X that the preemption wording was "not precise enough" and would be revised, and that the bar for blocking a model cannot be "so high that the power never gets used when it's needed" (The Next Web, 20 October).

Why this matters to me

On 25 February 2023 I wrote a short law of chatbots. Its central clause was a gate: no developer releases a chatbot to the public until an International Authority for Chatbots Approval has seen the draft and issued either an "R" certificate, for research institutes, or a "P" certificate, for general public use. I set out the rest of the text clause by clause in Vindicated: Parekh's Law of Chatbots, and I returned to it in April this year, when Mythos was held back, and asked for a pause on unrestricted release of frontier cyber-capable models until a certification regime existed.

Put my 2023 clauses beside Anthropic's draft and the family resemblance is not subtle.

What I proposed, February 2023What Anthropic's draft does
Submit the draft chatbot to an approving authority before releaseEngage a registered independent evaluator at least every 180 days; the findings bind no one
Release only with a certificate: R for research, P for publicNo certificate. One binary question: is there an imminent risk of 50 deaths or $1 billion in damage?
Duties on output, with human feedback and built-in controls (clauses A to H)Safety framework, transparency report at each deployment, incidents reported within 15 days, security audits
Enforced by the certifying authorityEnforced only by Commerce, only in a D.C. federal court, 45 days at a time

That is real movement. Nothing in US statute today lets any regulator stop an AI model from shipping; California's SB 53, the strongest state law on the books, is a disclosure and incident-reporting regime. A bill that makes independent assessment a condition of doing business at the frontier, and writes a blocking trigger into law, is a different category of proposal from what the industry has offered so far.

Then it makes three design choices I would take apart.

Three choices I would take apart

1. The trigger measures harm, not capability

Fifty deaths and a billion dollars are the wrong unit of measurement. They are the vocabulary of an inquest: they describe what has already happened, or what someone imagines might. In When Weapons Outsmart Us I argued for capability thresholds, sandboxes and staged rollouts, treating transformative AI systems like novel therapeutics: measurable technical benchmarks that trip a requirement for testing and authorization before deployment.

A capability threshold can be tested in a laboratory. A death toll can only be estimated by the developer who wants the estimate low, or by an evaluator guessing at scenarios. One is engineering. The other is forensics.

The narrowing makes it worse. The blocking pathway covers biological weapons but, per the aipolicy.tech reading of the text, not chemical, radiological or nuclear ones, even though California's definition, which the draft borrows its numbers from, covers all four. If you are going to legislate a threshold, legislate the one your own definitions imply.

2. The firewall is built out of the thing it is meant to stop

The draft permits a lab to pay the evaluator that audits it, so long as payment is not contingent on the finding. Anyone who has watched a supplier relationship knows which contingency matters. It is not this invoice. It is the next contract, the renewal, the referral. The draft sets no licensing standard for evaluators, who register and can certify themselves, and LawAI's Charlie Bullock told Transformer that this would set auditors competing to offer the least demanding checks at the lowest price.

My 2023 proposal was naive about many things. It was not naive about this. The body that issues the certificate has to be an authority, not a vendor.

3. The kill switch is judicial

Under the draft, Commerce must file in the US District Court for the District of Columbia, prove its case by a preponderance of the evidence, and renew the order every 45 days. The court decides "without deference" to Commerce or to the evaluators, and it must weigh the harm an order would do to the developer, to users and to the public. Transformer notes that it is not told to weigh the opposite harm, of letting the model run.

In the drug analogy I keep returning to, the FDA does not petition a judge to suspend a trial. The aviation regulator does not ask a court before grounding an aircraft. Catastrophes do not observe court listing, and a model already serving millions of users does not stand still while a docket moves.

The defence is due process: an agency that can switch off a company's product on its own say-so is a real danger too, and a judge is a check on caprice. I accept that argument in general. I do not accept it as an answer to a 72-hour clock.

Where it goes further than the rest

The honest counterweight is that this still goes further than the industry's loudest position. Nvidia's chief executive, Jensen Huang, has argued that the AI industry does not need new laws at all (TechCrunch, 20 October). Anthropic's draft accepts mandatory independent evaluation, incident reporting, security audits and a federal blocking power, and it creates a legal trigger where none existed. A badly shaped gate is still a gate, and gates change behaviour, because developers plan releases around them.

My complaint is not that the draft exists. It is that if this becomes the template, the template will be a post-mortem trigger, a conflicted inspector and a court calendar. Progress, badly shaped, is the fairest verdict I can give it.

What this means in India

The numbers have already travelled. The power has not.

California's SB 53, signed on 29 September 2025, defines catastrophic risk in exactly the same units: more than 50 deaths or serious injuries, or more than $1 billion in damage to property, from a single incident in which a frontier model helps build a chemical, biological, radiological or nuclear weapon, commits an unsupervised cyberattack or serious crime, or evades control (California Business and Professions Code, chapter 25.1). Indian readers should notice what SB 53 does with that definition. It demands a published safety framework, incident reports, whistleblower protection and civil penalties. It lets nobody stop a model. The metric is travelling even where the statute is not, and Delhi will meet the phrase "50 deaths or one billion dollars" in somebody's consultation paper soon enough.

Import the certificate, not the courtroom

An Indian lab selling into the American market will meet this regime before it meets an Indian one, because the draft keys coverage to training compute, not to where a company is registered. The threshold is set at 10 to the power of 25 operations, which aipolicy.tech calls GPT-4-era compute, low enough that anyone fine-tuning a large open-weight model arguably becomes a frontier developer. Budget for an evaluation you do not control, and read the blocking clause as carefully as the safety clause.

What to do now

These are recommendations, most urgent first.

  1. This week, read the statutory text rather than the summaries: the blocking section, the evaluator section, the preemption clause. Three pages decide the argument.
  2. If you are a frontier developer, publish your own capability thresholds now. That shifts the debate from "what body count?" to "what test?", which is the ground you want it fought on.
  3. If you are an evaluator, write a fee policy that refuses future work tied to findings, and publish it. Independence that is not documented will not be believed.
  4. If you are a legislator, in Washington or in Delhi, separate the three things inside this bill: the duty to test, the duty to report, the power to stop. Each can be improved on its own timetable.
  5. Longer term, push for the graded certificate. A research-only release followed by a public one is what my R and P certificates were for, and nothing in this debate has replaced them.

What not to do: do not treat "50 deaths or $1 billion" as a safety standard. It is a jurisdictional threshold for a court case, and it will read like a licence for everything below it.

The bigger picture

Two trends are worth watching. First, the regulated are now drafting the regulation, and the regulator's job is turning into the negotiation of a company's discussion draft. Anthropic has already promised to rewrite its preemption language after criticism, which is what accountability looks like when the author is also the lobbyist. Second, regulation is drifting from capability to incident: it is easier to legislate a number than a test.

My prediction is that the preemption clause shrinks and the judge stays. The thing to watch is not whether this bill passes. It is whether the FRONTIER Act, which puts the emergency power in an agency rather than a court, keeps that authority when the two texts are merged.

Sources

Posted by

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)