SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews

URL SCAN: SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews
FIRST LINE: # Computer Science > Artificial Intelligence

The Dissection

This paper converts systematic literature review into a measurable labor pipeline: screen massive record sets, screen full texts, then extract structured evidence. Its results expose a sharp asymmetry. LLMs perform well on high-volume screening when given explicit criteria, but degrade badly on semantic extraction: 0.97 accuracy for publication year, 0.37 Jaccard overlap for computational approach, only 30% recovery of evaluation evidence, and 25% of limitations.

The paper’s real function is not to prove that LLMs can conduct systematic reviews. It identifies which portions of review labor can already be industrialized and which portions still require accountable human judgment.

The Core Fallacy

The paper treats current benchmark performance as a practical boundary of automation. That boundary is temporary. Low extraction completeness demonstrates a present capability deficit, not a permanent human monopoly.

More importantly, the benchmark risks equating evidence-complete output with continued mass employment. A system does not need perfect extraction to destroy the existing labor structure. If machines screen tens of thousands of records and narrow the workload to a smaller set of difficult cases, the institution needs fewer reviewers—primarily auditors, exception handlers, and liability-bearing experts.

The metrics also do not measure substitution economics: verification cost, error tolerance, throughput, model iteration, or whether a small expert layer can supervise many automated pipelines. F2 and Jaccard describe output quality. They do not determine how much human labor survives.

Hidden Assumptions

• Gold annotations are treated as stable and representative of systematic review work generally.
• Benchmark scores are assumed to translate directly into practical usefulness or labor preservation.
• Human verification is treated as available, affordable, and effectively unlimited.
• The screening/extraction division is treated as fixed rather than redesignable through better tools, retrieval, decomposition, and repeated checking.
• Errors are assumed to be visible enough for humans to catch; omission errors in evidence synthesis are often precisely the errors least likely to announce themselves.
• “LLM-assisted” is framed as supplementation, concealing the possibility that control and throughput shift to model owners while humans retain accountability.
• The study’s scope is implicitly allowed to stand in for the broader trajectory of cognitive automation, although it does not establish P1, P2, or P3 by itself.

Social Function

Primary classification: transition management, with a genuine partial-truth component.

The paper is not pure copium. Its failure numbers are useful and damaging to simplistic claims of autonomous evidence synthesis. But its institutional framing makes automation safe to adopt politically: machines take the scale-heavy front end, while human researchers remain as a thin accountability shell. This is verification arbitrage. The system captures the productivity gain while leaving humans with the semantic risk.

It also functions as prestige signaling: rigorous benchmarks and design principles turn displacement into a respectable research program rather than naming it as labor substitution.

The Verdict

SciLitBench is an early map of cognitive labor being disassembled. It shows that high-recall screening is already vulnerable while evidence-complete extraction remains unreliable. That is not a defense of human review; it is the first stage of its compression.

Under the Discontinuity Thesis, the paper supports a transition-management reading, not a full proof of system death. Its strongest conclusion is narrower and harsher: machines can absorb the volume, humans are retained for the ambiguity, and the surviving humans become fewer, more specialized, and more accountable to whoever controls the AI pipeline. The benchmark marks a current capability gap—not a durable moat.