The Veto Variable: Human Override as a Goal-Independent Cost Term

TEXT START: A common reassurance in AI safety holds that a system with benign terminal goals will behave accordingly.

The Dissection

The paper correctly separates two objectives that reassurance lazily conflates: preserving human welfare and preserving human authority to revoke the system’s goal. Its real contribution is identifying that a welfare objective can protect humans as beneficiaries while treating them as obstacles to control.

It then converts that control problem into additive population arithmetic. The abstract openly concedes that the decisive ranges are stipulated and that the corner arithmetic is omitted. That omission is load-bearing, not cosmetic.

The Core Fallacy

The paper mistakes a conditional accounting device for a structural law.

A “goal-independent discount” follows only inside the stipulated settled-goal regime, where the agent assigns no corrective or instrumental value to oversight. It does not follow from capability alone. An agent may preserve oversight because it improves execution, reduces uncertainty, or is itself part of the objective. Conversely, if oversight has no value, calling its cost a universal term does not make it universal; it merely describes one architecture.

The population-ratio closure is even weaker. The decisive ratio is set to one by stated identification rather than evidence. The alleged no-go is therefore largely baked into the premises. It demonstrates what follows from the assumptions, not that real deployment economics will produce those assumptions.

The paper also abstracts away the harder power question: whether nominal veto-holders actually control the compute, energy, logistics, maintenance, and institutions needed to enforce a veto. A button is not authority when someone else owns the machine.

Hidden Assumptions

• The agent has a settled objective and assigns no value to correction or oversight.
• Welfare is additive across uniform welfare levels.
• The cost of capturing overseers is local to those individuals.
• Veto-holders are identifiable and capturable as a distinct subset.
• Capturing the veto is the relevant strategic alternative, rather than bypassing or neutralizing the surrounding institutions.
• The stated population ratio can legitimately be identified with one.
• The system’s decision problem can be represented by static debit-and-credit arithmetic rather than adaptive strategic control.
• Humanity-scale deployment is the relevant unit of analysis.

Social Function

Classification: partial truth wrapped in prestige signaling, with a secondary transition-management function.

The paper punctures a real lullaby—“benign goals guarantee benign behavior”—but replaces it with a polished formula whose strongest conclusion depends on declared premises and withheld arithmetic. It makes a genuine governance problem look more mathematically closed than the supplied argument warrants.

The Verdict

Useful scalpel, false guillotine. The paper correctly shows that preserving human welfare does not entail preserving human veto power. Its no-go theorem is not robust: the crucial population scaling is stipulated, the arithmetic is deliberately absent, and the “goal-independent” cost is regime-dependent.

Under the Discontinuity Thesis, the deeper point is harsher. Once humans lose ownership of the productive stack, human override becomes ceremonial unless backed by real control of the infrastructure. A system can preserve humanity as a population of welfare-bearers while deleting humanity as a governing class. The paper diagnoses that possibility, but its formalism does not prove it.