LLMs learn scientific taste from institutional traces across the social sciences

URL SCAN: arXiv preprint | cs.AI | v3, May 2026
FIRST LINE: "Reinforcement-learned reasoning has powered recent AI leaps on verifiable tasks, including mathematics, code, and structure prediction."

THE DISSECTION

This paper automates the last cognitive redoubt that academia claimed was irreducibly human: taste. The operation is precise: take the historical record of what peer review permitted through gates controlled by fallible, biased, often petty humans → train a 4-billion-parameter model to replicate those decisions → celebrate that the machine now outperforms the humans who made the original decisions.

The "scientific taste" being learned is not some Platonic ideal of quality floating free from human institution. It is the residue of historical gatekeeping — what a specific set of journals, editors, and reviewers at specific moments decided was publishable. Training on publication outcomes trains on bias, power, fashion, and exclusion as much as on any genuine epistemic signal.

The paper frames this as progress. It is. Just not in the direction the authors intend.

THE CORE FALLACY

The paper smuggles in the assumption that historical publication decisions are a legitimate proxy for research quality. They are not. They are a proxy for:

• Conformity to disciplinary fashion at the moment of review
• Social network effects in peer selection
• Resource asymmetry between elite and peripheral institutions
• Cumulative advantage in citation economies
• Gender, racial, and geographic bias in gatekeeping

Teaching an LLM to replicate these outcomes does not produce an improved evaluator. It produces a fossilized snapshot of historical power relations rendered into weights. The paper notes, without apparent alarm, that the model learned "scientific taste" from this mess. That is not a feature. That is the mechanism by which institutional dysfunction gets automated and locked in.

HIDDEN ASSUMPTIONS

1. Peer review as quality signal — The paper treats publication outcomes as ground truth. This is circular: it uses what passed through gates to teach machines how to run those gates.

2. Disciplinary legitimacy as given — Eight "social science disciplines" are treated as natural kinds rather than institutional constructions with their own politics, fads, and exclusions.

3. Prediction of publication = prediction of value — The paper conflates "will this get past reviewers" with "is this scientifically valuable." These are orthogonal. Some of the most important work in history was initially rejected.

4. Human expert judgment as baseline to beat — Framing human experts as the standard implies their failure is evidence of model superiority rather than evidence that the evaluation task itself may be incoherent or corrupted.

5. Calibration as virtue — "Calibrated confidence" is presented as evidence of genuine evaluative skill. An AI can be perfectly calibrated while systematically wrong. Accuracy is the only metric that matters; the paper's emphasis on calibration is a rhetorical maneuver to claim sophistication beyond mere pattern-matching.

SOCIAL FUNCTION

Classification: Prestige signaling + institutional automation play

This is academic AI research doing what academic AI research does: taking a human institution, showing it can be automated, and presenting the automation as scientific contribution. The authors are not villains. They are performing their role in a system that rewards "we built a thing that works better than humans at X" papers.

The paper's enthusiasm for its results reveals the underlying ideology: institutional traces as legitimate signal. The entire premise assumes that what got published is what should have been published. This is the epistemic core of academic conservatism — the existing distribution of prestige is treated as evidence of merit, and now machines are being trained to perpetuate it.

Secondary function: transition management. By demonstrating that "taste" is learnable, this paper participates in the broader project of demonstrating that no cognitive domain is safe from automation. Each such demonstration normalizes the next.

THE VERDICT

What this paper actually does: Teaches a small language model to predict which research proposals will survive human gatekeeping, by training on the historical record of gatekeeping decisions. The model beats humans because human gatekeeping is noisy and inconsistent — not because the model has found some deeper truth about scientific value.

What this means for the DT framework: Another domain of human evaluative judgment falls. "Scientific taste" was supposed to be the irreducible remainder — the craft knowledge, the disciplinary intuition, the tacit understanding of what matters that no algorithm could capture. This paper demonstrates otherwise, with quantitative rigor that will make it cited as evidence that even the softest cognitive work is automatable.

The structural implication: Scientific institutions face a choice between using AI to perpetuate their existing biases at scale or attempting to rebuild evaluation on explicit, auditable criteria. They will choose the former because it is cheaper and because "what we did before" is the only foundation most reviewers know how to stand on. The result: a scientific culture that trains its own replacement gatekeepers on its own historical exclusions, then calls the outcome "taste."

The paper is technically competent and epistemically hollow. It is an autopsy dressed as a progress report — a demonstration that the peer review system is learnable precisely because peer review is not a clean signal but a noisy historical artifact. Training on that artifact does not produce a better evaluator. It produces a more scalable reproducer of whatever the artifact encoded.

The authors do not seem to understand what they have found.