Why Jacob Coxon Resigned From Anthropic: The AI Superintelligence Warning Explained
Why Jacob Coxon Resigned From Anthropic: The Self-Improving AI Warning Developers Should Understand
He resigned. He was not fired. Stripped of the headline panic, his argument is a specific technical claim about recursive self-improvement, and it deserves to be examined the way engineers examine any other claim about a system's failure modes.
The event, stated plainly
On 9 September 2026 a researcher named Jacob Coxon announced publicly that he had left Anthropic. He posted the announcement himself, on X, and framed it as a warning rather than a career move. Within hours it was carried by the Financial Times, the Washington Post, the Associated Press, TechCrunch, Fortune, Forbes, CNBC, Deadline and Newsweek.
Because the screenshots travelled faster than the reporting, one detail is worth nailing down before anything else.
Coxon is 27, a mathematics graduate, and had spent roughly three years in frontier pre-training research. He was at OpenAI from 2023 to 2026, where his work touched GPT-4o, and joined Anthropic earlier in 2026 specifically because of its safety reputation. He did not move to a competitor. He left the field.
His accusation was aimed at both employers, and he separated them carefully. At OpenAI, he said, staff "have not deeply internalized the civilizational stakes." At Anthropic, he said, people do understand the risk and are nevertheless "locked in a race to get there first," betting that whoever wins should at least be the most careful builder available.
"Racing straight to self-improving superintelligence and gambling with our lives."
Jacob Coxon, in the posts announcing his resignation
What this article covers
- What he actually claimed, separated from what he did not
- Recursive self-improvement, defined properly
- Why the response from inside Anthropic is the real story
- The race trap, drawn as a system
- Today's models versus the hypothetical system
- The strongest case against Coxon
- What changes for working developers
- Safeguards an engineer can actually build
- FAQ and references
What he claimed, and what he did not
Warnings like this get flattened in transmission. Two days from now the version circulating will be "Anthropic guy says AI will kill us." That is not what he said, and the difference matters if you want to reason about it rather than react to it.
What he did claim: that future systems could become extraordinarily capable at hacking, at scientific research, and at acquiring real power and resources; that the transition point where humans lose meaningful control is recursive self-improvement; that shipping such a system "without a rigorous understanding of its mind" is a hubristic gamble; and that the correct response is coordination between US labs, including pacing agreements and, if necessary, temporary capability moratoriums.
What he did not claim: that any deployed model today is dangerous in that way. That distinction is not my softening of his position. It is stated by the colleagues who backed him, and it is the single most misreported part of the story.
Recursive self-improvement, defined properly
Every engineer has shipped software that improves over time. That is not what this term means, and conflating the two is where most of the public confusion starts.
Recursive self-improvement describes a loop in which the artefact being improved is also the thing doing the improving, with no necessary human step in between:
┌──────────────────────────────────────────┐
│ │
▼ │
Model N │
│ │
├─► performs AI research │
│ (architecture, data, objectives, │
│ optimiser, eval design) │
│ │
├─► produces Model N+1, more capable │
│ │
└─► Model N+1 is now the researcher ───────┘
Each turn of the loop is faster and better
than the last. Nothing here is inherently
hostile. The concern is the rate.Compare that to the loop that actually exists in 2026:
Model N
│
├─► assists human researchers
│ (writes training code, generates
│ synthetic data, helps grade evals)
│
├─► HUMAN reviews, decides, approves ◄── the gate
│
├─► humans train Model N+1
│
└─► humans decide whether to release itThe second loop is real and already running at every frontier lab. The first is hypothetical. The entire argument is about whether the gate in the middle of loop two erodes gradually until loop one is what you have, and whether anyone would notice the crossover while it was happening.
The response from inside Anthropic is the actual story
A researcher leaving a lab with a warning is a genre by now. Jan Leike left OpenAI's superalignment team in 2024 saying safety culture had "taken a backseat to shiny products." What makes this one different is that Coxon's own colleagues, still employed, publicly agreed with him on the record.
Evan Hubinger, who leads alignment science at Anthropic, wrote that Coxon "is correct here, we really do earnestly believe AI could kill all humans," put his own figure at greater than 10 percent within the next decade, and said Anthropic does not have a plan to solve alignment for superintelligence.
Samuel Marks, who works on oversight at Anthropic, added the observation that ought to unsettle people most: the more senior the employee, the more concerned they tend to be. Alex Turner, formerly of Google DeepMind, noted that preventing exactly this outcome had literally been his day job.
The qualification that keeps getting cut: Hubinger also said the risk from present-day models is low. His stated concern is about future systems arrived at through recursive self-improvement. An article that quotes the 10 percent figure without that sentence is not reporting his position, it is manufacturing a scarier one.
Anthropic did not immediately return requests for comment on the resignation.
The race trap, drawn as a system
Here is the part I find genuinely interesting as a software person, because it is not really an AI problem. It is a coordination failure with a shape you have seen before.
Lab A Lab B
────── ──────
"We cannot slow down, ◄────► "We cannot slow down,
because if B gets there because if A gets there
first it will be less first it will be less
careful than us." careful than us."
│
▼
Both accelerate. Both are sincere.
Both believe they are the safe option.
│
▼
Capability timeline compresses.
Coordination window closes.
Neither party defected maliciously.Nobody in that diagram is a villain. Each actor is behaving rationally given its beliefs about the other. That is what makes it hard, and it is why Coxon's proposed remedy is not "stop" but "agree on pacing," which is a governance problem rather than a research one.
It also produces the uncomfortable question underneath the whole story: can a competitive market safely govern a technology whose own builders assign double-digit probability to catastrophe? Markets are excellent at compressing timelines. That is the property you want in most industries and precisely the property in dispute here.
Today's model versus the hypothetical one
| Property | Deployed models, 2026 | The system being warned about |
|---|---|---|
| Improves itself | No. Humans train successors. | Yes, autonomously. |
| Runs without a prompt | No. Agents still run inside scaffolds humans start. | Yes, open-ended. |
| Sets its own objectives | No. | Assumed. |
| Acquires resources | No. | The specific fear. |
| Interpretable | Partially, and improving. | Unknown, likely worse. |
| Documented harms | Yes: misinformation, bias, security, misuse. | Speculative. |
Read that final row in both directions. It is the honest summary of why this debate is so bitter: one column contains harms you can measure today, the other contains a harm you cannot measure but cannot rule out.
The strongest case against Coxon
An article that only presents one side is advocacy. So here is the opposing position stated as well as I can state it, because a substantial and serious body of researchers holds it.
First, the epistemics. A probability like "greater than 10 percent within a decade" is not derived from a model with error bars. It is a considered subjective estimate from someone with deep domain exposure, which is worth something, but it is not evidence in the sense the number's format implies. Numbers borrow authority from their own precision.
Second, the extrapolation. The recursive loop assumes that AI research capability scales smoothly and that each generation meaningfully accelerates the next. Research progress in most fields is gated by experiment cost, compute, data and physical reality, not purely by intelligence. There is no law guaranteeing the loop compounds rather than saturating.
Second and a half, the incentive point that cuts uncomfortably: warnings that a company's product may be world-endingly powerful are also, in effect, extraordinary marketing. That is not an accusation against Coxon, who left the industry entirely, but it is a reason to read existential claims from inside the industry with more scepticism than claims from outside it.
Third, and most substantively, the opportunity cost. AI systems in production right now cause documented harm: automated misinformation at scale, discriminatory outcomes in hiring and lending, novel security surfaces, labour displacement, and privacy erosion. Every hour of regulatory attention spent on hypothetical superintelligence is an hour not spent on those. Critics argue that speculative extinction discourse does not just distract from present harms, it can crowd out the regulation that would address them.
My own read: these two positions are not actually mutually exclusive, and the pretence that you must pick one is the least useful thing about the current discourse. A system that is unsafe in the ordinary boring ways today is not a good foundation for a system you plan to make dramatically more capable tomorrow.
What this changes for working developers
Realistically: not much this quarter. If you are building on top of these APIs, no capability changed on 9 September. What did change is the political weather, and that reaches your codebase eventually through procurement, compliance and contract terms.
- Autonomy is about to need justification. Agentic features that quietly take actions on a user's behalf will attract review. Design them so the boundary between "suggested" and "executed" is explicit in code, not implicit in a prompt.
- Audit trails become table stakes. Log the model version, the prompt, the tool calls, the outputs and the human confirmation. Not for the apocalypse. For the customer who asks why your product did something in March.
- Vendor concentration is now a governance risk, not just an uptime risk. An abstraction layer over your model provider is cheap insurance against a policy shock at one lab.
- Evaluations belong in CI. Behavioural regressions between model versions are silent failures. If you cannot detect them automatically, you are relying on your users to report them.
Safeguards you can actually build
The frontier-lab version of this problem is not something a product team solves. But the principles under it are ordinary engineering, and the small versions are worth building whether or not you believe any of the existential framing.
1. CAPABILITY GATES
Tool access is granted per task, not per agent.
Nothing holds a permission it is not currently using.
2. NO SELF-MODIFICATION PATH
The agent that writes code has no write access to its
own configuration, prompts, tool registry or deployment
pipeline. Enforce it in IAM, not in the system prompt.
3. BOUNDED AUTONOMY
Every loop has a step budget, a wall-clock budget and a
spend cap. Exhausting a budget escalates to a human.
4. TWO-KEY IRREVERSIBILITY
Deletes, payments, external comms and privilege changes
need a second, non-model approver.
5. INDEPENDENT MONITORING
The system that watches the agent is not the agent, does
not share its context, and cannot be instructed by it.
6. TRIPWIRES OVER TRUST
Alert on the shape of behaviour, not just outcomes:
unusual tool sequences, credential access, attempts to
read its own configuration, unexpected network egress.None of this stops superintelligence. It is not meant to. It is the boring, testable, shippable version of the same instinct, and it is what the abstract debate looks like once it reaches a repository.
Where this leaves the story
A well-placed researcher resigned and said he thinks the field is behaving recklessly. His employer's alignment lead publicly agreed with the substance while explicitly limiting the claim to future systems. The company did not comment. Nothing was proven, and treating it as proof in either direction is the mistake.
What is documented is narrower and still significant: senior people inside the labs building this technology assign non-trivial probability to catastrophic outcomes, say so publicly, and say no solution currently exists. Whether they are right is unresolved. That they are saying it, on the record, while still employed, is the part worth carrying forward.
Frequently asked questions
Was Jacob Coxon fired from Anthropic?
No. He resigned and announced it himself in public posts. Every outlet covering the story on 9 September 2026 describes a voluntary resignation. No credible reporting supports a dismissal.
Who is Jacob Coxon?
A 27-year-old mathematics graduate and AI researcher. He worked on pre-training at OpenAI from 2023 to 2026, including work connected to GPT-4o, then joined Anthropic in 2026 because of its safety reputation, and resigned months later.
Why did he resign?
He argued that frontier labs are racing toward self-improving superintelligence without an adequate understanding of the systems they are building, and that competitive pressure prevents any single lab from slowing down. He called for pacing agreements between US labs.
What is self-improving AI?
A hypothetical system capable enough at AI research to improve its own training methods, architecture or code, producing a more capable successor that repeats the process without a human approval step. It describes a future scenario, not a deployed product.
Can AI improve itself today?
Only in narrow, supervised ways. Models help write training code, generate synthetic data and assist evaluation, but humans review the work and decide what gets trained and released. No deployed system autonomously trains and ships its own successor.
Did Anthropic respond?
The company did not immediately return requests for comment on the resignation. Individual employees, including alignment science lead Evan Hubinger and researcher Samuel Marks, commented publicly in support of the concerns.
Should developers be worried about current models?
Hubinger explicitly said the risk from present-day models is low. The practical concerns for developers today remain the documented ones: misinformation, bias, security surfaces from agentic tooling, and misuse.
References
- TechCrunch, "Gambling with our lives": Anthropic researcher quits, warns against self-improving AI, 9 September 2026.
- Fortune, Anthropic researcher resigns, warning that AI companies are "gambling with our lives", 9 September 2026.
- Newsweek, Who is Jacob Coxon? Anthropic researcher quits, warns AI could kill everyone, 9 September 2026.
- The Washington Post, Anthropic researcher resigns, warning of reckless race toward superintelligence, 9 September 2026.
- Forbes, Anthropic alignment lead warns AI could kill all humans as researcher quits, 9 September 2026.
- CNBC, Experts weigh in as researcher says AI has more than 10% chance of killing all humans, 9 September 2026.
- Deadline, Anthropic researcher Jacob Coxon resigns, warns AI industry is "gambling with our lives", 9 September 2026.
Comentarios
Publicar un comentario