Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models

URL SCAN: Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models
FIRST LINE: LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures - incorrect answers and confident hallucinations - both rooted in the same cause...

TEXT ANALYSIS: THE DISSECTION

This is a 2026 arXiv paper documenting a controlled experiment. The finding: when you give LLMs a small (4KB) hand-authored semantic document describing business logic, accuracy on text-to-SQL tasks jumps 17-23 percentage points. The models tested (Claude Opus 4.7, Sonnet 4.6, GPT-5.4) become statistically equivalent with the document, and statistically equivalent without it. Model choice within tier doesn't matter. Only the presence of explicit semantic metadata matters.

What it's really doing: Producing a rigorous demonstration that the current LLM stack for enterprise analytics has a fixable interface problem. The models aren't failing because they're too dumb—they're failing because the business domain knowledge that humans encode in their heads isn't encoded anywhere the model can read. The paper correctly diagnoses the symptom: forced semantic inference. The proposed fix: externalize those semantics as a structured context layer.

The Core Fallacy (DT Lens): The paper treats this as a reliability engineering problem—something to be patched so LLM-powered analytics can be trusted in production. This framing treats the underlying trajectory as stable. The paper does not ask: reliable for what, and for whom, and for how long?

The semantic layer is essentially human knowledge digitization work. Someone has to write the 4KB document. Someone has to know the business semantics well enough to encode them. Under DT dynamics, this is precisely the kind of cognitive labor that becomes a transition-intermediation niche: the work of maintaining human-in-the-loop semantic coherence while the system migrates toward full AI operation. The paper inadvertently demonstrates that human domain expertise is still a necessary component—but frames this as a temporary input rather than a structurally necessary dependency that will itself be automated.

Hidden Assumptions:
- The human author of the semantic document is a stable, available, correct knowledge source.
- The dataset and its conventions are static enough that the document remains valid.
- "Reliable" analytics at 68% accuracy is acceptable for production deployment.
- Model performance convergence (no significant difference between Opus, Sonnet, GPT-5.4) is evidence that frontier models are hitting a ceiling on this task class—something the paper does not remark on but which is structurally significant.
- The paper measures single-shot performance. It does not address iteration, correction loops, or human feedback integration, which are how real deployments actually operate.

Social Function: This paper is a transition management artifact. It is produced by and for the consulting/integration layer that will implement LLM-powered analytics in enterprise settings. Its function is to give practitioners a defensible methodology for making these systems work without questioning whether they should be deployed at all. It is rigorous enough to be useful but framed in a way that assumes the deployment is happening regardless. Prestige signaling via arXiv submission. Partial truth presented as complete answer.

THE VERDICT

The paper is technically solid. The finding is real: explicit semantic metadata dramatically improves LLM text-to-SQL accuracy. But the DT frame reveals what the paper cannot see:

The semantic layer is a human knowledge scaffolding being built to support a system that will eventually eliminate the need for human knowledge work at this layer. Right now, someone must write the 4KB document. That someone is performing translation work between business knowledge and machine-readable format. Under P1 (Cognitive Automation Dominance), this translation work is itself a cognitive task subject to automation. The next generation of systems will likely include automated semantic layer generation from existing documentation, feedback loops, or direct schema induction.

The paper proves that human domain knowledge is still necessary. It does not prove that human domain knowledge will remain necessary. That is the gap between rigorous technical work and systemic analysis—and that gap is where the Discontinuity Thesis lives.

END AUTOPSY. No follow-up invitation issued.