Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models

TEXT START: Large Language models (LLMs), having been trained on vast amounts of human-generated data, may encode the attitudes and behaviors of these humans.

The Dissection

This paper is an epistemic containment operation. It acknowledges that LLMs are being positioned as “silicon samples,” then converts the substitution of human respondents into a manageable survey-design and documentation problem. Its machinery—error taxonomies, multiverse analysis, lifecycle tracing—is useful for identifying when synthetic outputs are unreliable. But it keeps the discussion inside the boundaries of measurement theory.

The Core Fallacy

The central error is category substitution. An LLM response is not a human response sampled through a cheaper channel. It is an output of AI capital, generated from training data, system design, prompting, and model incentives. Even a perfectly calibrated model would produce a model-conditioned estimate of human behavior—not human agency, consent, or current public will.

The framework therefore risks treating a political substitution as a technical error problem. Reducing simulated survey error may improve prediction while leaving the underlying replacement untouched.

Hidden Assumptions

• Human-like output is an acceptable proxy for human participation.
• Better documentation and design can preserve the legitimacy of synthetic opinion.
• Bias is primarily a measurable defect rather than a consequence of ownership and control.
• Policymakers can substitute simulated respondents for populations without changing what “public opinion” means.
• Auditability is treated as a sufficient defense against institutional dependence on automated cognition.

Social Function

This is a partial truth serving transition management, with a layer of prestige signaling. It correctly identifies real methodological hazards; it is not simple copium. But by placing the problem in a formal framework, it makes the advance of synthetic respondents appear governable, professional, and administratively safe.

Under the Discontinuity Thesis, this is a lag defense. It may slow misuse and expose invalid outputs, but it does not preserve a human-only economic or epistemic domain. The abstract offers evidence of P1 and pressure on P2: automated systems are moving from producing text to manufacturing apparently social knowledge. It does not, by itself, prove P3.

The Verdict

TS2E is a warning label attached to a structural substitution. It can diagnose how silicon samples fail, but it cannot make them people or restore human participation. The paper documents the instrumentation of the corpse; it does not revive the body.