What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks

TEXT START: Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reasoning, refusal).

THE DISSECTION

This paper is an autopsy of leaderboard epistemology. It shows that benchmark labels are not reliable descriptions of stable abilities. Safety benchmarks sharing a concept often produce weakly related rankings, while capability benchmarks labeled differently can behave almost identically. Design artifacts—especially score format—may explain performance better than the supposed construct.

BBQ-accuracy is the cleanest wound: labeled as a bias benchmark, it correlates more strongly with reasoning benchmarks. The instrument is not measuring what its name claims. The paper therefore exposes benchmarks as political and institutional measuring devices whose apparent precision exceeds their validity.

THE CORE FALLACY

The paper correctly attacks benchmark validity but remains trapped inside the psychometric enclosure. It assumes that convergent and discriminant validity are the decisive tests of whether a capability matters. Under the Discontinuity Thesis, the decisive question is different: does a measure predict autonomous task completion, labor substitution, coordination power, and control of AI capital?

A benchmark can be construct-valid yet economically irrelevant. Conversely, a messy benchmark can still reveal movement toward cognitive automation. Correlations among model rankings do not establish causal latent abilities, and improved benchmark hygiene cannot repair the employment-to-consumption circuit once AI severs it. It can only make the dashboard less deceptive while the engine continues replacing human labor.

HIDDEN ASSUMPTIONS

• That assigned concepts such as “reasoning,” “knowledge,” “bias,” and “refusal” are coherent enough to serve as common categories.
• That correlation among model rankings is evidence of shared underlying capability rather than model scale, training data, contamination, prompting, or evaluation format.
• That 53 models provide a representative basis for general conclusions about model behavior.
• That benchmark scores and item responses are sufficiently comparable for cross-benchmark inference.
• That item-response theory assumptions hold across heterogeneous datasets, tasks, and safety constructs.
• That benchmarks can isolate capabilities which, in deployment, interact and compound.
• That better measurement will materially improve governance rather than merely refine institutional reporting.
• That naming the construct correctly is close to understanding its economic consequence. It is not.

SOCIAL FUNCTION

Classification: partial truth, transition management, and prestige signaling.

The paper punctures benchmark theater, which is valuable. But it channels the critique into a more sophisticated evaluation apparatus—validity tests, ranking correlations, IRT models, and a larger dataset. That gives institutions a respectable way to demonstrate methodological seriousness while postponing the harder question: who owns the automated productive system, and what happens when most people are no longer economically necessary?

It is not pure copium. It identifies genuine measurement failure. Its anesthetic effect begins when benchmark repair is mistaken for systemic repair.

THE VERDICT

This is a strong instrument audit and an incomplete transition analysis. It demonstrates that AI benchmarks are semantically unstable, vulnerable to design artifacts, and often incapable of supporting the claims attached to them. It does not weaken P1, P2, or P3. If anything, it reveals a recognition lag: institutions are still arguing over whether the gauges measure “reasoning” or “bias” while the underlying automation mechanism advances.

The scoreboards are cracked. That does not mean the machine is failing. It means the observers are losing the ability to tell which parts of the machine’s progress are real.