Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing

URL SCAN: Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing
FIRST LINE: Computer Science > Computation and Language

The Dissection

This paper is an empirical autopsy of AI detection. It shows that screening is trained on a frozen target while the production system keeps mutating. At a 1% false-positive rate, detectors can fall from catching over 99% of rewrites before a model boundary to catching only 3.8% after it. Across all 23 versions, institutions must choose between falsely flagging one in eight human abstracts or missing one in three rewrites of the newest model. This is not a calibration nuisance. It is a control system losing lock on its target.

The Core Fallacy

The paper treats a structural legitimacy failure as a maintenance problem. Re-verifying a detector after every release does not create a stable human-authored category; it creates an infinite compliance treadmill. Later-trained detectors can miss earlier models, and vocabulary is the primary transfer signal. There is no final detector state.

More fundamentally, detection of machine-shaped prose is not proof of authorship, intellectual contribution, or scientific validity. The paper’s procedure confuses policing a textual surface with preserving human productive participation. Under P1–P3, that participation is precisely what cognitive automation is removing.

Hidden Assumptions

• Journals can obtain and test every relevant model version quickly enough to keep pace.
• A small, finite benchmark remains representative of future model behavior.
• False positives and missed rewrites can be traded off without destroying institutional legitimacy.
• Stylistic classification can legitimately substitute for provenance, accountability, or proof of contribution.
• Institutions can enforce human-authorship norms while AI is embedded throughout the research workflow.
• Updating the detector preserves the underlying screening regime rather than merely extending its administrative half-life.

Social Function

Classification: partial truth, transition management, and ideological anesthetic.

The empirical finding is real and damaging: detector benchmarks decay at model-generation boundaries. But the policy recommendation gives research-integrity institutions a procedural patch—retest after every release—so they can continue the screening ritual while pretending the instability is manageable. Attention stays on catching forbidden prose instead of confronting the larger fact that automated writing is eroding the scarcity of human cognitive labor.

The Verdict

The paper does not rescue AI screening. It proves that the detector layer is a lagging bureaucracy attached to a faster production system. Its recommendation may preserve false certainty temporarily, but it cannot restore authorship scarcity, stable enforcement, or mass productive participation. The detector is not a moat; it is a smoke alarm whose batteries are replaced after each fire. Under the Discontinuity Thesis, durable leverage belongs to the Sovereigns who control the writing systems and the Servitors who verify, govern, and maintain their outputs—not to institutions trying to preserve the old order through version-by-version policing.