Why So Many AI Researchers Think the Machines Could Kill Everyone

TEXT START: Earlier this year, Rishub Jain left his position as an artificial intelligence researcher at Google DeepMind after a revelation.

The Dissection

This is a panic memo from inside the engine room. It documents frontier researchers discovering that recursive self-improvement could turn AI development into a feedback loop humans no longer understand or control. The resignations, containment failures, cyber incidents, and biolab scenarios serve as evidence of capability outrunning governance.

But the article frames the crisis primarily as a question of whether AI might exterminate humanity. That is the cinematic endpoint. The more immediate structural event is economic: AI severs the mass employment → wage → consumption circuit, while ownership of the productive systems concentrates among the firms and people controlling AI capital. The article sees the machine’s teeth but focuses on the possibility of a bite to the skull while ignoring the systematic removal of human economic necessity.

The Core Fallacy

The central error is confusing uncertain extinction risk with the more mechanically predictable collapse of productive human participation.

Recursive self-improvement is not established by this text; it remains theoretical. But the article does establish that firms are competing toward increasingly autonomous systems despite incomplete oversight, weak alignment guarantees, and powerful incentives to continue. That is enough to activate the Discontinuity Thesis.

Human-in-the-loop oversight is not equivalent to human control. If AI performs the analysis, coding, evaluation, and strategic work, the human becomes a ceremonial approval layer unless they own or command the system. Alignment may reduce the chance of hostile behavior. It does not preserve wages, bargaining power, or productive indispensability. A safe machine can still make most humans economically obsolete.

The article also treats quitting, safety startups, and coordinated slowdown as meaningful brakes. Under competitive dynamics, they are mostly individual acts of conscience inside a race that rewards whoever ignores the brakes. P2 remains intact: institutions cannot reliably preserve human-only economic domains at scale once superior automated cognition is available.

Hidden Assumptions

• That human oversight can scale with systems whose complexity already exceeds practical comprehension.
• That safety work can outrun capability work while drawing funding from the same competitive machine.
• That researchers quitting meaningfully constrains the labs replacing them.
• That the main definition of “harm” is extinction, cyberwar, bioweapons, or mass violence.
• That keeping a human nominally involved preserves actual agency.
• That alignment solves the ownership and distribution problem.
• That coordination among labs and states can remain stable despite enormous strategic advantages for defectors.
• That uncertainty justifies continued experimentation rather than exposing who controls the resulting productive assets.

Social Function

Partial truth, elite self-exoneration, ideological anesthetic, and transition management.

The article permits insiders to confess that the project may be dangerous while preserving the legitimacy of continuing it. It turns a structural competition into a morality play about brave researchers, safety funding, and better oversight. That creates the appearance of responsibility without confronting the ownership question: who receives the output when machines perform the work, and what happens to everyone excluded from productive participation?

It also provides a useful escape hatch for the industry. If catastrophe arrives, executives can claim they warned the public. If it does not, they can claim the warnings justified their safety programs. Either way, the race continues.

The Verdict

The article is credible about one danger and strategically blind about the larger one. It shows that AI capability is advancing faster than human control, but mistakes human extinction for the primary system failure.

Under the Discontinuity Thesis, the decisive threat is not whether machines develop a dramatic desire to kill humanity. It is that humans voluntarily build systems that no longer need most humans economically, then leave ownership concentrated in the hands of the few who control them. The extinction scenario is a tail risk. Productive participation collapse is the central mechanism. The post-WWII order does not need to be murdered by a rogue superintelligence; it can be rendered obsolete by obedient machines.