Not Every Call Needs a Frontier Model: Per-Call-Site Evaluation of Small Language Models in a Deployed Agentic Home-Automation System

TEXT START: An agentic system issues several structurally different kinds of LLM calls.

The Dissection

This is an economic decomposition disguised as a model benchmark. It splits one agent into five purchasable functions—routing, classification, grounding, planning, and code generation—and tests how little intelligence each requires. Four of five sites are already locally commoditized. Frontier capability survives only at code generation, a narrow bottleneck.

The result is more consequential than the paper’s modest framing suggests: agentic automation can be made cheaper by routing calls to adequate models instead of paying frontier rates for every step. The 91.8% routed accuracy versus 95.4% at full cost, and the live result of 39/43 at 28% of the spend, show that inference expense is already a weak barrier to deployment.

The Core Fallacy

The paper’s error is not necessarily bad measurement. It is treating task-level substitutability as an engineering optimization instead of a power transition. It measures whether a model can perform a call, not whether humans retain bargaining power once cognitive tasks become cheap, modular, and owned by whoever controls the system.

Under the Discontinuity Thesis, this is not evidence that human work remains valuable. It is evidence that cognitive labor can be atomized and priced downward. The code-generation gap is a lag defense, not a durable moat. Once coding is further decomposed, scaffolded, or verified by machines, the final premium is attacked by the same routing logic.

Hidden Assumptions

• The five-call taxonomy remains stable as systems improve.
• Two Home Assistant installations and 280 cases represent broader deployment conditions.
• A 3.6-point accuracy loss is acceptable across all error types.
• Refusal and false actuation are comparable outcomes. They are not: unauthorized actuation is an asymmetric safety failure.
• Per-call spend captures the relevant cost, excluding integration, maintenance, liability, security, energy, and oversight.
• The live user score of 39/43 captures rare or catastrophic failures.
• Code generation remains the enduring human-adjacent bottleneck.
• The human remains economically central merely because a human still supplies requests.

Social Function

Primary classification: partial truth serving transition management, with an ideological-anesthetic effect.

The paper is not empty copium. Its routing result and actuation warning are real. But it converts a civilization-scale substitution process into managerial language about model selection and spend. It leaves ownership invisible: who controls the models, device graph, energy, infrastructure, and liability? The user remains a user, not a productive participant.

The Verdict

Technically, this is a useful benchmark. Systemically, it is an early autopsy of the cognitive labor price floor. It provides concrete support for P1 in a deployed domain: most agentic work does not require frontier intelligence, so automation can become cheap and scalable.

It does not prove P2 or P3; two homes cannot establish institution-wide coordination impossibility or mass productive-participation collapse. But it points directly toward both. The frontier model is not the killer. The routing layer is.