Two missing pieces in the AI safety discussion

TEXT START: This was the week that AI safety hit the big time.

The Dissection

This is a persuasion brief for AI “pacing,” not a complete safety analysis. It makes three moves: converts abstract extinction probabilities into a bioweapon scenario; treats AI leaders’ public fear as evidence of sincere concern; and proposes cyberattacks using Chinese models to frighten the CCP into slowing development.

Its strongest point is that software capability can be coupled to physical systems, laboratories, supply chains, and malicious actors. But the article does not demonstrate the full causal chain from model capability to viable pathogen design, physical access, release, global spread, and extinction. It substitutes a plausible narrative for proof.

The essay also identifies the Red Queen’s race while offering no mechanism capable of stopping it. The proposed “demonstration attack” is not safety policy. It is escalation gambling.

The Core Fallacy

The central error is turning “AI might gain a dangerous capability” into “covertly attack China so its leaders will impose restraint.” Every decisive step is assumed rather than established:

• Capability becomes reliable biological or cyberweapon performance.
• Performance becomes existential threat.
• Threat becomes Chinese recognition.
• Recognition becomes cooperation.
• Cooperation becomes enforceable global restraint.

The final step is especially weak. An attack on Chinese infrastructure could produce acceleration, retaliation, secrecy, repression, or militarization—the opposite of the intended result. The article recommends creating the very crisis it claims must be prevented.

Its dismissal of biologists is also intellectually lazy. The claim that experts underestimate AI because humans have been surpassed in other domains is a warning against complacency, not evidence that pathogen design is tractable. “Experts may be wrong” cannot carry the burden of proof for an extinction policy.

The deeper Discontinuity Thesis failure is category error. The article treats biological extinction as the decisive danger while ignoring P1–P3: cognitive automation, coordination failure, and the collapse of economically necessary human labor. Humanity can remain biologically alive while the postwar economic order dies. Pacing may delay that transition; it cannot restore mass productive participation or solve ownership of AI capital.

Even the supposed evidence of elite sincerity is ambiguous. A slowdown can protect incumbent laboratories by raising barriers to entry and freezing their existing advantage. Sincerity is not proof of analytical correctness, and public fear is not proof that their preferred remedy serves the public.

Hidden Assumptions

• “Superintelligence” and recursive self-improvement are imminent, measurable conditions.
• A model can design a catastrophic pathogen and overcome all biological, manufacturing, access, and containment constraints.
• A catastrophic outbreak would become human extinction rather than severe but limited devastation.
• Chinese leaders would correctly attribute an attack to Chinese models and interpret it as an internal dissident threat.
• The CCP would fear domestic AI-enabled revolt more than it values strategic and military advantage.
• A cyberattack could remain controlled, non-escalatory, and free of civilian consequences.
• The United States could conduct such an operation without triggering retaliation or an arms race.
• Frontier labs can coordinate and enforce a slowdown despite competitive pressure, open models, diffusion, and strategic rivalry.
• Public alarm will produce competent regulation rather than surveillance, incumbent capture, or authoritarian control.
• Extinction risk deserves priority over the quieter but structurally certain displacement of human labor.

Social Function

Classification: partial truth, elite self-exoneration, transition management, ideological anesthetic, and—especially in the China section—propaganda.

The partial truth is real: AI may amplify biological and cyber risks, and competitive dynamics make voluntary restraint unstable. But the essay casts lab executives as trapped custodians rather than participants in the race producing the danger. It redirects public attention toward spectacular extinction scenarios while leaving ownership, labor displacement, and power concentration largely untouched.

“Pacing” therefore functions as elite transition management: slow the machinery enough to buy institutional adaptation while preserving the existing command structure. The proposed attack on China adds a more dangerous function—manufacturing fear in an adversary and hoping it behaves predictably.

The Verdict

Smith correctly identifies a real tail risk and correctly sees that AI competition is structurally difficult to restrain. His remedy is defective at the point where it matters most. A covert cyberattack intended to induce Chinese restraint could accelerate the race it claims to stop.

Under the Discontinuity Thesis, this is a sharp alarm about one possible apocalypse, not an analysis of the system’s actual terminal mechanism. It warns about extinction while ignoring economic obsolescence. Its “pacing” program is hospice care: it may buy lag, but it cannot resurrect the mass employment–wage–consumption circuit.