Should we "pace" AI self-improvement?

TEXT START: A guest post by Tim Fist and Saif Khan.

The Dissection

The article converts a potentially uncontrolled capability race into a governance problem. It identifies automated AI R&D as a possible accelerator of biosecurity threats, loss of control, and power concentration, then proposes “pacing” as a calibrated policy lever.

Its central move is administrative: define danger thresholds, redirect resources, improve verification, build resilience, and preserve U.S. advantage. This treats recursive self-improvement as something institutions can monitor and selectively throttle while still harvesting broad social benefits.

The text also conspicuously avoids the larger discontinuity. It discusses catastrophe and concentration, but not the destruction of the mass employment–wage–consumption circuit. “Diffusing AI capabilities” is treated as broadly beneficial even though labor-replacing diffusion accelerates the collapse of productive human participation.

The Core Fallacy

The article assumes that AI R&D automation can be separated from the competitive system that makes it economically and strategically mandatory.

If automated AI R&D produces superior capability, firms and states have incentives to pursue it. A stable human-controlled ceiling therefore requires coordination across competitors, borders, and potentially autonomous systems. That is precisely the coordination impossibility the Discontinuity Thesis identifies. Pacing is not a durable middle path; it is a lag defense imposed on actors rewarded for violating it.

The second fallacy is treating safety, diffusion, and resilience as substitutes for ownership and control. They may reduce particular risks, but they do not restore mass productive participation. A society can avoid a supervirus and still suffer systemic death when AI capital performs the work and ownership remains concentrated.

Hidden Assumptions

• Dangerous levels of automated AI R&D can be defined and measured before they cause harm.
• Governments can distinguish safe from unsafe research without creating false confidence or exposing strategic information.
• Frontier firms and states will accept enforceable limits despite overwhelming competitive incentives to defect.
• Global coordination can prevent rogue firms, adversarial states, open-source systems, or autonomous agents from bypassing the limits.
• Time bought through pacing will actually produce robust defenses rather than simply extend the capability race.
• Safety research will keep pace with capability growth.
• AI diffusion will distribute power rather than amplify the owners of compute, models, energy, and infrastructure.
• “Low-regret” policies will remain low-regret when the underlying technology is moving faster than institutions can evaluate it.
• U.S. strategic dominance will create control and stability rather than intensify geopolitical escalation.
• Human oversight will remain meaningful after systems begin contributing materially to the design of their successors.

Social Function

This is a partial-truth transition-management document with strong elements of elite self-exoneration and ideological anesthetic.

It is not pure copium: it correctly identifies acceleration, offense-dominant capabilities, oversight failure, and power concentration as serious mechanisms. But it relocates the political crisis into technical thresholds and state capacity. That lets laboratories and policymakers signal responsibility while preserving the underlying race and ownership structure.

“Pacing” functions as reputational insurance: continue building the machine, promise to slow it if necessary, and present preparation as control. The proposal also frames the U.S. lead as a safety asset, quietly assuming that concentration of power can be used to manage concentration of power.

The Verdict

The article sees the fuse but imagines a committee can meter its burn.

Pacing may buy time, delay specific catastrophes, or create temporary regulatory moats. It cannot reverse P1, overcome P2, or prevent P3 if AI achieves durable superiority across cognitive work. At best, it is hospice care for the post-WWII order: useful for managing the transition, useless for preserving the system that made mass employment economically central.

The decisive question is not whether governments can slow automated AI R&D. It is who controls the resulting AI capital—and who remains indispensable once it does.