Skill-based Agentic Evaluation for Real-time Data Science Tasks

URL SCAN: Skill-based Agentic Evaluation for Real-time Data Science Tasks
FIRST LINE: Computer Science > Artificial Intelligence

The Dissection

This paper builds a testing apparatus for agents that query changing data. Its central move—ground-truth-as-code—turns expected answers into executable functions, then compares agent outputs against recomputed atomic facts rather than stale prose references. The 29% MCC improvement and 16% token reduction indicate better evaluation efficiency within the tested environment.

The deeper function is industrialization: it converts live data-science work from an expert judgment problem into a measurable, repeatable, machine-scored production process. That is not protection for data scientists. It is the scaffolding required to replace more of them.

The Core Fallacy

The framework risks confusing verifiability with intelligence. If an answer can be specified as executable ground truth, the task is already partly formalized and therefore unusually exposed to automation. The evaluator proves that an agent produced the expected computation; it does not prove that the agent selected the right question, understood ambiguous business context, detected corrupted inputs, challenged invalid premises, or recognized that the reference function itself encodes the wrong objective.

Ground-truth-as-code solves reference staleness. It does not solve reality staleness. Live data can remain current while being incomplete, manipulated, misclassified, or semantically misunderstood. A continuously updated wrong answer is still wrong—just more efficiently certified.

Hidden Assumptions

• The expected answer can be expressed as executable code without losing the important judgment in the task.
• The reference function, schemas, entity relationships, and data permissions are correct.
• Atomic fact decomposition captures correctness across prose, tables, HTML, and other formats.
• An LLM judge can reliably score precision and recall without introducing a new, less visible failure layer.
• A synthetic database that reproduces production structure adequately represents production ambiguity, noise, access failures, and adversarial conditions.
• Better benchmark agreement transfers to open-ended, shifting, multi-source work.
• The tasks selected for evaluation represent data science broadly rather than the tractable slice that is easiest to formalize.

These assumptions define a narrow corridor of competence. Inside it, the agent can be measured. Outside it, the benchmark has little authority.

Social Function

Primarily transition management and partial truth, with a layer of prestige signaling. The paper truthfully identifies a real bottleneck: static benchmarks decay when the underlying data changes. It then supplies the measurement layer needed to deploy cognitive agents into live operational systems.

Its social effect is larger than its stated evaluation goal. By making agent performance legible, auditable, and cheaper to score, it removes one of the institutional excuses for keeping human analysts in the loop. The paper is not announcing the death of data science. It is manufacturing the gauges, test harnesses, and compliance artifacts required before that death can be operationally trusted.

The Verdict

This is competent infrastructure for the Discontinuity, not a defense against it. It attacks evaluation uncertainty—the final bureaucratic brake on agent deployment—while leaving the human labor circuit untouched because its purpose is to make that circuit less necessary.

Under P1, executable ground truth makes a growing class of data work cheap to automate. Under P2, organizations will not preserve large human-only domains once agents can be continuously tested against live operational truth. Under P3, the remaining human role contracts toward exception handling, ownership, governance, and system maintenance.

The paper’s real achievement is therefore not proving that agents are intelligent. It is making their failures countable enough for institutions to buy them anyway. That is how cognitive labor becomes a managed cost center before it becomes a casualty.