Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

TEXT START: Language models are increasingly used to create synthetic survey samples, also called silicon samples.

The Dissection

This paper is not really asking whether machines are human. It is asking whether machines can reproduce the observable output of human survey labor. It separates aggregate sample similarity from prediction of an individual respondent and finds that the latter fails while the former is already competitive with respondent-derived statistical baselines.

The important findings are blunt: the best generative model approaches the performance of a Gaussian copula fitted to other respondents; probability vectors preserve human item marginals better than rounded answers; and profile fields add little to predicting an individual’s responses. The model does not need to understand a person if the buyer only needs distributions, correlations, rankings, or plausible population estimates.

The Core Fallacy

The central error, under the Discontinuity Thesis, is treating person-level psychometric fidelity as the decisive threshold for replacement. That is the wrong battlefield. Economic substitution does not require an authentic inner life. It requires a cheaper system that produces sufficiently useful outputs.

The paper’s own recommendation—to optimize for sample-level fidelity—is more operationally important than its title suggests. A synthetic respondent can be psychologically invalid and still make the human respondent economically unnecessary. The machine need not replace the person; it only needs to replace the paid output.

Hidden Assumptions

• PSS adequately measures practical validity rather than merely resemblance to one survey’s response structure.
• The Lithuanian organisational-psychology survey and its 263 respondents generalise to other populations, instruments, and decision contexts.
• Aggregate errors will average out rather than concentrate in minority groups, tails, or high-impact cases.
• Human responses are the relevant ground truth, rather than external behaviour, later outcomes, or causal validity.
• The buyer wants statistical plausibility more than provenance, individual authenticity, or legally defensible participation.
• The weak performance of profile regression applies broadly; the data only establish that these profile fields reveal little for this instrument.
• Model regressions across product releases represent temporary noise rather than a permanent obstacle to competitive automation.

The 100-person comparison also exposes metric fragility: at that sample size, PSS cannot reliably distinguish real respondents from top models. That weakens the paper’s ranking claims, but it does not restore the human moat. It makes the boundary between authentic and synthetic output harder to police.

Social Function

Classification: partial truth and transition management.

The paper punctures the fantasy that models can reconstruct individual human psychology. Simultaneously, it legitimises synthetic respondents as sample-level instruments and converts the replacement of human participation into a calibration problem: measure similarity, retain probability vectors, select the best model.

This is not copium. It is the paperwork of transition. Human respondents are being repositioned from routine production inputs into calibration data, validation references, edge-case suppliers, and sources of legal or institutional cover.

The Verdict

“Probable but Not Valid” is accurate only if validity means person-level psychological identity. Under the Discontinuity Thesis, that standard is economically excessive. The decisive result is that aggregate synthetic samples are already near simple respondent-based baselines while individual profile information contributes almost nothing.

The paper does not prove universal survey automation; one narrow instrument cannot establish that. It does reveal the mechanism: once buyers need statistically serviceable outputs rather than genuine human participation, the respondent’s productive role is exposed as a replaceable surface. The remaining human demand—ground truth, unusual cases, external validation, and institutional legitimacy—is lag defense and transition labor, not a reversal of the machine’s advantage.