Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities

TEXT START: A deployed LLM agent emits tool calls, queries, and code that can be silently wrong -- by the time the error surfaces, the action has run.

The Dissection

This is an operational control layer for black-box agents. It converts a cheap surrogate’s log-probabilities into a triage signal: identify dubious argument values, judge whole calls, compare competing tools, then gate or feed the score back into the agent.

The paper’s real product is not confidence. It is cheaper oversight. It makes autonomous tool use more deployable by reducing the human attention required per action and by catching failures before they compound.

The Core Fallacy

The narrow technical claim is credible within the supplied results: a surrogate can detect some failures better than the actor’s own stated confidence or naive resampling.

The systemic limitation is treating improved auditability as containment of automation. It is not. A better smoke detector does not reverse the fire. It lowers the cost of operating the machine while it burns through human cognitive work.

Under the Discontinuity Thesis, this strengthens P1. It also undermines the remaining justification for human execution: once agents can act, detect their own weak points, and request review selectively, institutions have less reason to retain humans across the full workflow. The human role contracts from producer to scarce exception handler. That is transition management, not preservation of the postwar labor circuit.

The word “confidence” also oversells the signal. These are model-relative error proxies, not truth probabilities. The reported AUROC gains demonstrate discrimination on the tested tasks; they do not establish calibration, robustness under distribution shift, resistance to correlated surrogate errors, or acceptable false-negative costs in consequential environments.

Hidden Assumptions

• The surrogate’s judgments remain correlated with the actor’s failures across models, tasks, tools, and changing prompts.
• A low-cost prefill pass remains cheap relative to the value and risk of the action.
• The surrogate sees enough context to detect errors without inheriting the same blind spots.
• Reviewers can absorb escalated calls, and review itself is faster and more reliable than autonomous execution.
• Benchmark accuracy and task-success gains transfer to real systems with adversarial inputs, ambiguous schemas, and irreversible actions.
• Confidence feedback improves the agent rather than giving it another misleading scalar to rationalize.
• Tool-choice competition works when the correct tool is absent, poorly specified, or constrained by organizational policy.
• Institutions will use the signal to allocate oversight rather than to manufacture a thin compliance veneer.

Social Function

Primary classification: transition management, with a substantial partial-truth component.

The technique is not mere copium. It solves a real engineering problem and makes silent agent failure more measurable. Its social function, however, is to make black-box autonomy tolerable to managers, regulators, and deployment teams. It supplies the audit plumbing required to move from “the model may hallucinate” to “the system can act at scale with selective human review.”

If marketed as human control, it becomes ideological anesthetic. The control is statistical, delegated, and economically optimized. It does not preserve mass productive participation; it prices and minimizes supervision.

The Verdict

Technically useful. Structurally accelerative.

This paper does not save the human-centered economic order. It helps convert unreliable cognitive automation into governable cognitive automation. That distinction is fatal. Better detection, selective escalation, and feedback increase the number of economically viable actions machines can perform without continuous human involvement.

In DT terms: P1 is strengthened, P2 is only locally managed, and P3 advances. The paper is not a brake on obsolescence. It is the quality-control system that lets the replacement machinery run faster.