AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks

TEXT START: Computer-use agents are capable of completing complex tasks, increasing the use of automatic judges to determine success, either for training or for evaluation without human involvement.

THE DISSECTION

This paper identifies the verification layer as the weak joint in agentic labor. Long trajectories can look complete while violating instructions or creating hidden side effects, so the paper turns trust failure into a benchmarkable engineering problem. Its practical function is to build the inspection apparatus required before autonomous cognitive labor can be deployed at scale.

THE CORE FALLACY

Relative to the Discontinuity Thesis, the implicit fallacy is treating unreliable judgment as the primary obstacle to automation. Even a perfect judge would not preserve human productive participation; it would make replacement safer and more scalable. The reported ceiling—80.9% balanced accuracy for the best judge on the benchmark subset—also means the control layer remains materially porous. This is evidence of immature automation, not evidence that human labor is structurally secure.

HIDDEN ASSUMPTIONS

• Screenshots and action histories contain enough evidence to establish real task completion and detect consequential side effects.
• User instructions can be interpreted consistently enough for paired trajectories to represent meaningful success and failure.
• Benchmark performance will transfer to unfamiliar applications, longer tasks, and higher-stakes environments.
• Judge errors can be reduced faster than the economic pressure to deploy imperfect agents grows.
• Human auditors remain available as a fallback when automated judgment fails.
• Balanced accuracy is an adequate proxy for deployment risk, despite asymmetric costs between false approval and false rejection.

SOCIAL FUNCTION

Partial truth and transition management, with a secondary prestige-signaling function. The paper is not copium: it openly documents serious reliability failures. But it confines the crisis to an engineering bottleneck, making the transition appear governable through better benchmarks, tools, and judges. It supplies the inspection machinery for the replacement economy rather than challenging replacement itself.

THE VERDICT

AgentHorizon is a diagnostic of the control layer, not a defense of the employment order. It shows that agentic labor still needs reliable adjudication before it can operate without supervision. That is a temporary moat for human overseers and a direct research roadmap toward eliminating them. Under the Discontinuity Thesis, the paper records friction in the arrival of P1—not a reprieve from P1, P2, or P3.