Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models

TEXT START: Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values.

The Dissection

The paper imports stated-preference economics as a quality-control system for language-model outputs where no answer key exists. It tests whether models produce theoretically coherent responses: downward-sloping demand, sensitivity to scope, and income effects. This is useful measurement of behavioral structure. It is not a discovery of truth.

The deeper function is to convert “unscorable” model behavior into auditable performance metrics. That makes subjective outputs legible to institutions, evaluators, and deployers. The paper therefore builds an evaluation regime for the post-ground-truth economy, where models increasingly generate judgments rather than retrieve facts.

The Core Fallacy

The central error is not that the authors confuse coherence with correctness; they explicitly warn against doing so. The error is more structural: they treat validity testing as though it substantially solves the evaluation problem when it only relocates it.

Economic theory supplies the tests, but the theory itself is not ground truth. A model can satisfy demand monotonicity, scope sensitivity, and income responsiveness while producing preferences that are fabricated, normatively distorted, or behaviorally useless. The framework can certify that an answer resembles a coherent economic agent without establishing that the agent, valuation, or recommendation corresponds to any real-world welfare consequence.

Under the Discontinuity Thesis, this is verification arbitrage: replacing unavailable truth with cheaper tests of consistency. That is commercially powerful and epistemically limited. The machine does not become an oracle because it obeys the shape of a theory. It becomes a more convincing synthetic respondent.

Hidden Assumptions

• Human stated-preference methods are treated as an adequate template for evaluating synthetic preferences, despite their own dependence on contested models, incentives, framing, and survey design.
• Theoretical regularities are assumed to remain meaningful when the respondent is a language model with no income, household budget, physical preferences, or autonomous consequences.
• Convergent validity is treated as informative even when models may share training data, prompting artifacts, and inherited human stereotypes rather than independently tracking reality.
• A response that passes validity tests is assumed to be useful for decision-making, although consequentiality may only reveal whether outputs influence behavior, not whether they deserve influence.
• Better evaluation is presumed to improve human control. Under P1 and P2, it may instead accelerate substitution by making automated judgment easier to certify and deploy.
• The absence of ground truth is framed as an evaluation inconvenience rather than a structural condition of an economy increasingly populated by machine-generated preferences and recommendations.

Social Function

Primary classification: transition management and prestige signaling, with a partial truth at its core.

The partial truth is substantial: many high-value model outputs cannot be graded against a single correct answer, and coherence, reliability, incentives, and consequences are legitimate dimensions of evaluation.

The transition-management function is more consequential. The framework gives institutions permission to operationalize model judgment before they can establish whether the judgment is actually valid in the world. It turns epistemic uncertainty into a dashboard. That is how adoption proceeds: not by solving the truth problem, but by making uncertainty administratively tolerable.

The paper also functions as elite self-exoneration in a mild technical form. It acknowledges that passing tests does not prove correctness, thereby disclosing the limitation, while still furnishing the machinery needed to institutionalize the outputs. The caveat becomes a liability disclaimer attached to an expanding automation pipeline.

The Verdict

This is a serious evaluation framework for measuring whether models produce internally structured preferences. It is not a framework for establishing that those preferences are true, welfare-improving, or socially legitimate.

Its importance under the Discontinuity Thesis is precisely that it helps cognitive automation cross the next barrier. Once synthetic judgments can be tested for reliability without ground truth, institutions no longer need to wait for certainty before replacing human evaluators, survey analysts, policy advisers, and recommendation workers. The paper does not defeat the automation thesis. It supplies part of its quality-assurance layer.

The models that pass are not validated minds. They are increasingly competent generators of economically actionable coherence. That distinction is the remaining human moat—and this paper is helping narrow it.