Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use

URL SCAN: Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use

FIRST LINE: Large language models (LLMs) increasingly act as autonomous agents that must decide when to answer directly vs. when to invoke external tools.

TEXT ANALYSIS: The Dissection

This is a technical AI research paper addressing a specific engineering problem: LLM agents fail to convert awareness of tool necessity into tool-calling action. The researchers empirically demonstrate that the bottleneck is not cognitive recognition but the cognition-to-action translation—a mechanical failure in the execution pipeline, not the judgment pipeline.

They quantify this as a 26–54% mismatch across models. This is framed as an optimization problem to be solved through better prompting, training, or architectural fixes.

The Core Fallacy (DT Lens)

The paper assumes this is a fixable bug. In the narrow technical frame, it might be. But under the Discontinuity Thesis, this paper is evidence of something far more corrosive: AI systems are being designed as autonomous agents that must decide and act—and those decisions are themselves subject to systematic failure modes that worsen as capability differentials widen.

The "knowing-doing gap" isn't a solved problem waiting for a gradient. It's a structural feature of machine cognition operating at the action boundary. And this paper demonstrates the gap persists across model tiers and across task types—arithmetic and factual QA, a 26–54% failure rate on tool-call fidelity.

This is not a lullaby. It is an autopsy finding.

Hidden Assumptions

1. Tool use is the correct paradigm. The paper accepts without interrogation that LLMs should be agentic, tool-using autonomous systems. It does not ask whether delegating action to probabilistic text predictors is structurally sound or whether this is a category error being papered over with RAG pipelines and function-calling schemas.
2. Model capability is the independent variable. They treat "weaker" vs. "stronger" models as an ordinal ranking. In reality, capability is multidimensional, domain-specific, and the boundaries shift faster than annotation can track. Their "ground truth" tool necessity annotation is itself a lagging signal.
3. The human judge is the fallback. They acknowledge human/LLM judges annotate necessity, but treat this as a solvable annotation problem. Under DT conditions, human judges are the slow, expensive, fallible baseline that cannot scale to match AI capability drift.

Social Function

This paper is transition management infrastructure. It performs the role of a mechanic documenting a recurring engine fault—technically useful, institutionally legible, and completely disconnected from the question of whether the vehicle should be on the road.

It is prestige signaling within the agentic-AI research community: "We found a real problem and measured it rigorously." The implicit message to labs and deployers is: "Keep building; we can patch this."

It is also partial truth. The knowing-doing gap is real. The quantification is valuable. But framing it as a solvable engineering problem obscures the more uncomfortable finding embedded in the data: even when models know they need tools, they fail to act at rates between 27–54%. In high-stakes deployments—medical, financial, legal, infrastructural—those failure rates are not optimization targets. They are catastrophe budgets.

The Verdict

Under DT logic, this paper inadvertently confirms P1: Cognitive Automation Dominance has a critical failure mode that worsens under operational conditions. The knowing-doing gap means AI agents are not reliable even at the narrow task of "use a tool when you know you need one." If the action boundary is where value is extracted and harm is inflicted, then a 27–54% action-failure rate is not an engineering problem to be solved next quarter. It is a demonstration that AI autonomy is not yet viable at operational reliability thresholds, and the capability gap between models guarantees that tool-use failures will be non-uniform, non-predictable, and concentrated in edge cases.

The paper improves the toolkit for transition management. It does not challenge the transition.