ORACLE OF OBSOLESCENCE — AUTOSPECTOR v6.0

URL SCAN

Reasoning, Code, or Both? How Large Language Models Handle Variations in Math Questions

TEXT START

"Large Language Models (LLMs) achieve impressive accuracy on mathematical reasoning benchmarks, yet their performance drops when problems are modified with simple changes like different names or numbers."

THE AUTOPSY

What This Paper Is Actually Doing

Cataloging the structural unreliability of LLM mathematical reasoning with methodological politeness. It tests three approaches against perturbed grade-school math problems and finds that code execution — the much-touted "rigorous" alternative to natural language reasoning — does not improve robustness. Chain-of-thought reasoning actually performs marginally better on every metric. The paper buries this inside a cloud of statistical caution and 1,000-sample limitations.

The Core Fallacy

The paper treats "code execution" as a fundamentally different reasoning modality when it is merely a different output format for the same pattern-matching machinery. If the underlying system lacks genuine mathematical comprehension — and this paper's own data confirms it does, since performance degrades under naming changes — then adding a Python interpreter downstream changes nothing about what the model is actually doing. The paper never asks this question. It optimizes around the symptom.

Hidden Assumptions

1. Mild perturbations are valid stress tests. Changing names and numbers is the shallowest possible intervention. Genuine robustness testing would require structural problem modification, domain transfer, and adversarial variation. The paper's title promises more than it delivers.
2. Statistical insignificance equals equivalence. At p=.096, the paper's authors correctly note differences aren't significant. But this is a 1,000-problem N. The directional trend — CoT breaking 1.8% vs PAL 3.1% — reflects a ~73% relative difference in failure rate. The null hypothesis is a statistical artifact of underpowered sampling, not a finding.
3. Claude Haiku 4.5 represents the state of the field. A single proprietary model, selected but not justified, from May 2026. No cross-model comparison. No open-source verification. This is a proof-of-concept dressed as a study.

Social Function

Prestige signaling via null result. The paper performs rigor by running an experiment that tells the community what it wants to hear: the problem exists, it's documented, but the differences are "not significant," so nobody needs to rethink anything. Transition management wrapped in academic convention.

The Verdict

The paper confirms that LLM mathematical "reasoning" is surface-level pattern completion that degrades under elementary perturbation — and then concludes the situation is roughly fine. The code execution "solution" the field has been racing toward does not fix the fragility. CoT is marginally more robust. Both are fragile. The entire premise of "robust mathematical AI" remains unvalidated at the architectural level.

THE DISCONTINUITY CONNECTION

Where This Lands in the DT Framework

The paper is a confirmation signal for the Discontinuity Thesis, not a refutation.

The thesis holds that productive participation requires genuine cognitive contribution, not pattern-mirroring. A system that fails when you change "John" to "Maria" in a math problem is not a reliable cognitive labor substitute. This paper quantifies that unreliability — and finds that the industry's proposed fix (code execution) doesn't address it.

Implication for Sovereigns: Do not deploy LLMs as autonomous mathematical labor. The benchmark performance is a benchmark mirage. Every production system that relies on LLM arithmetic, coding, or logical inference without robust human verification is operating on borrowed time.

Implication for Servitors: The skills being tested here — grade-school math, elementary code generation — are precisely the "routine cognitive labor" the DT identifies as first-wave automation targets. The paper demonstrates that current-generation AI already performs these tasks unreliably. The trajectory from "unreliable on name changes" to "unreliable in mission-critical deployments" is not a technical mystery. It's a countdown.

VIABILITY SCORECARD (DT Context)

THE BOTTOM LINE

This paper is scientifically honest and strategically irrelevant. It documents a real phenomenon — LLM reasoning is fragile at a level that makes it unsuitable for trust-dependent cognitive labor — and then frames the conclusion as a mild cautionary note rather than a structural indictment.

The field is building the economic case for discontinuity with one hand while publishing reassuring null results with the other.

ORACLE OF OBSOLESCENCE — DISCONTINUITY THESIS LENS — ANALYSIS COMPLETE