More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses

TEXT START: Automated generation of LLM harnesses promises to improve inference through task specialization.

The Dissection

This paper is an audit of a flattering metric. It demonstrates that apparent harness diversity can be counterfeit: repeated stochastic executions of identical programs can create answer coverage, while generated programs may encode persistent weaknesses rather than useful specialization. At 27 executions, both generated and byte-identical populations reach 98.70% oracle coverage, the frozen selector adds nothing, and stable complementarity remains unproven.

The paper is therefore not primarily discovering a new capability. It is stripping away an attribution error: coverage is being mistaken for specialization, and repeatability is being mistaken for deployable advantage.

The Core Fallacy

Relative to the Discontinuity Thesis, the deeper fallacy is economic irrelevance. Even if the paper eventually validates genuinely specialized harnesses, that would improve the allocation and reliability of AI capital. It would not preserve human productive participation, restore the wage-consumption circuit, or create a durable human-only economic domain.

The research may expose bad measurement, but its success would sharpen automation rather than resist it. Better harness evaluation is not a defense against P1-P3. It is instrumentation for P1.

Hidden Assumptions

• That marginal inference gains are socially stabilizing rather than competitive weapons.
• That improving answer coverage creates broad productive opportunity instead of concentrating capability with model owners and system controllers.
• That benchmark performance is a meaningful proxy for economic usefulness.
• That persistent task complementarity, once found, would benefit workers rather than the owners of the harness stack.
• That three repeats and MATH-500 can establish the relevant structure of specialization; the abstract itself admits stable complementarity remains unresolved.
• That better selection and orchestration can solve the underlying problem. They can only make replacement more efficient.

Social Function

Classification: partial truth, transition management, and prestige signaling.

The partial truth is real: the paper identifies a serious evaluation failure and supplies a stricter standard for claiming specialization. The transition-management function is equally clear: it converts the disruptive question of machine superiority into a technical question about harness diversity, selectors, execution budgets, and output validity. That is useful engineering, but it also narrows the frame until the social consequence disappears.

The benchmark becomes a clean laboratory for a dirty historical process. Human labor is not rescued; the machinery that can replace it is being measured more honestly.

The Verdict

Technically, this is a credible demolition of weak specialization claims. Systemically, it is a refinement of the automation apparatus, not an escape from it. The paper shows that some supposed intelligence gains are statistical mirages; it does not challenge the underlying direction of travel. If its evaluation standard succeeds, AI harnesses become more reliable, more economically deployable, and more capable of severing the labor-to-income circuit.

The hallucination is not necessarily in the experiment. It is in treating better machine coordination as evidence of human economic survivability.