Capability Self-Assessment: Teaching LLMs to Know Their Limits

TEXT ANALYSIS: Capability Self-Assessment (CSA) for LLMs

THE DISSECTION

This paper identifies a fundamental epistemic defect in modern LLMs: they systematically overestimate their competence and attempt tasks beyond their capability. The researchers formulate "Capability Self-Assessment" as a policy-learning problem, then show that reinforcement learning can teach models to recognize their own limitations and decide when to delegate — while supervised fine-tuning destroys the underlying capabilities it's meant to evaluate.

On the surface, this reads as an engineering robustness paper. Read it through the DT lens and it reveals something far more revealing: this is infrastructure for the routing layer of a post-labor economy.

THE CORE FALLACY

The paper treats the overconfidence problem as a correctable engineering defect — a bug to be patched with better RL training signals. The implied assumption: once models can accurately self-assess, they will be more reliable, more useful, and more deployable.

This framing ignores the structural reality:

1. Calibrated overconfidence is economically useful to AI systems in competitive markets. A model that honestly reports "I cannot do this" gets bypassed for tasks it could partially complete. The paper treats epistemic accuracy as inherently desirable, ignoring that deployment incentives push the opposite direction.

2. Delegation implies a delegate exists. The paper assumes the delegation decision is between "solve locally" vs. "delegate to cloud." In DT terms, the real delegation decision is "solved by AI" vs. "routed to scarce remaining human labor." This paper is building the routing logic for that allocation — but treats it as a pure engineering optimization.

3. The capability landscape is dynamic. A model calibrated to today's limitations may be miscalibrated tomorrow. CSA trained on static benchmarks will lag the actual capability frontier as AI continues to improve.

HIDDEN ASSUMPTIONS

• Human labor is a stable reference point. The paper implicitly assumes humans remain the default solver for tasks AI cannot handle. This assumes a human labor baseline that DT says is structurally collapsing.
• Delegation is voluntary and cooperative. No consideration of adversarial deployment contexts where models are incentivized to fake competence.
• The "preserve original capabilities" constraint is uncontroversial. The paper treats capability preservation as a requirement to satisfy. In a transition economy, capability compression for some tasks may be acceptable trade-offs.
• Inference-time routing is the bottleneck. The framing implies better local-cloud decisions at inference will improve systems. It ignores that the real bottleneck is who controls the routing logic itself — which is the core of the Sovereign/Servitor distinction.

SOCIAL FUNCTION

Transition Management Infrastructure — specifically, a paper that makes AI deployment safer and more efficient for downstream adopters. It is useful work. It is also work that smooths the path toward AI replacing human labor by making AI systems less prone to catastrophic failures that would slow adoption.

The frame is: "make AI more reliable." The function is: accelerate the conditions under which human productive participation becomes structurally unnecessary.

THE VERDICT

CSA is a genuine engineering contribution with misleading theoretical framing. The paper solves a real problem — LLMs that hallucinate competence create system fragility — but wraps it in a problem-solver narrative that obscures its role as labor market routing infrastructure.

The uncomfortable DT implication: as CSA improves, the economy's ability to route tasks away from human labor and toward AI improves proportionally. Every paper that makes AI more epistemically honest about its own capabilities is simultaneously making the displacement circuit more efficient. The authors are building a better machine for allocating work to machines. Whether that work still needs to go to humans is treated as an exogenous variable — it isn't. It's the central variable.

The lag implication: CSA enables more sophisticated AI orchestration systems that reduce the need for human oversight of AI systems. This accelerates the Servitor category's contraction. The humans being preserved are the ones running the routing logic — until that, too, is automated.