OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences

URL SCAN: OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences
FIRST LINE: Computer Science > Artificial Intelligence

The Dissection

This paper is building a legitimacy machine for a future claim: AI can enter research territory where humans have not already supplied the answer. The 14.0% mean judged solve rate for GPT-6-Astra is significant as a capability signal, but it is still evaluator-mediated. Four models judged submissions without reference solutions. That measures plausible, structured progress—not independently verified theorem production.

The benchmark also selects 82 problems whose decisive claims admit comparatively clear checks. It therefore samples a tractable slice of open theoretical science, not the entire research frontier. The paper’s strongest evidence is that performance improves with better problem representation, arguments extending beyond finite evidence, and proofs of completion steps. That is more serious than retrieval or stylistic imitation. It is still not autonomous science.

The Core Fallacy

The implicit leap is from “AI can sometimes make progress on selected open problems” to “AI has achieved general scientific intelligence.” The supplied result does not establish reliable reproducibility, broad coverage, independent verification, long-horizon research ability, or economic cost superiority.

Under the Discontinuity Thesis, this benchmark neither proves nor disproves P1. It does not establish durable superiority across cognitive work, and it says nothing direct about P2 or P3. But it damages the human-exception defense: frontier theoretical reasoning is no longer safely classified as an exclusively human domain. A failure rate of roughly 86% does not preserve human economic centrality if machines are already cheaper and faster at enough adjacent cognitive tasks.

Hidden Assumptions

• Model evaluators can accurately judge novel mathematical or computational claims without reference solutions.
• “Degree of progress” is sufficiently objective to support a meaningful solve rate.
• The 82 selected problems represent the wider frontier rather than a favorable benchmark slice.
• A submission that appears to contain the needed proof steps will survive expert scrutiny and reproduce independently.
• Improvements from representation and prompting reflect general reasoning ability rather than benchmark-specific optimization.
• Performance on foundational theory transfers to economically consequential cognitive work.
• Human researchers remain the scarce source of validation rather than becoming a temporary verification layer around machine-generated work.

These assumptions are load-bearing. Remove them and the headline number becomes a confidence score attached to model outputs, not a census of solved science.

Social Function

Primary classification: prestige signaling and transition management, with a genuine partial truth.

The paper converts speculative AGI capability into a leaderboard, a solve rate, and a recognizable scientific arena. That helps institutions normalize AI as a participant in research before it is fully reliable. It is not pure copium: a model producing general arguments and proof steps on unresolved problems would represent real cognitive penetration. But treating 14.0% judged success as proof that science—or human scientific labor—is safe is ideological anesthesia.

The Verdict

This is a warning shot, not proof that the post-WWII order has already died. It shows AI beginning to erode one of the last prestige barriers: novel theoretical cognition. The benchmark is too selected and too evaluator-dependent to establish P1, P2, or P3. Its real significance is directional. If judged progress becomes independently verified, reproducible, and cheap at scale, the human researcher will be pushed into servitor or validator status unless they control the systems producing the work. The paper records the breach in the wall; it does not yet prove the fortress has fallen.