CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design

URL SCAN: CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design
FIRST LINE: Computer Science > Artificial Intelligence

The Dissection

CADWorld is an artifact-integrity stress test disguised as a computer-use benchmark. It moves evaluation beyond clicking through interfaces and asks whether an agent can produce a valid, persistent FreeCAD project: geometrically correct, parametrically structured, constraint-valid, manufacturable, simulatable, and technically documented.

The result is a clean exposure of the current weakness. A 17.5% best-agent success rate against an 87.0% expert reference pass means general GUI fluency is nowhere near reliable engineering autonomy. Agents are not merely making cosmetic mistakes; stronger systems still fail on the hidden skeleton of professional work—the construction process, relationships, constraints, and downstream state.

The Core Fallacy

The central category error is treating benchmark competence as equivalent to economic replacement, or treating present incompetence as evidence that replacement will not occur. CADWorld measures whether agents can complete a demanding workflow today. It does not measure the cost curve, improvement rate, deployment scale, verification overhead, liability allocation, or whether a smaller number of engineers can supervise vastly more machine output.

The benchmark therefore proves neither human immunity nor imminent extinction. It establishes something more useful: CAD automation is currently bottlenecked by persistent-state reliability, not by the ability to generate plausible screen activity. Once those structural failures are reduced, the economic question changes abruptly. A benchmark score need not reach 100% for human labor demand to collapse; it only needs to make expert supervision cheaper than maintaining the existing staffing pyramid.

Hidden Assumptions

• Executable artifact checks are treated as an adequate proxy for real engineering quality, although they may not capture design judgment, safety margins, manufacturability in unfamiliar contexts, or liability.
• FreeCAD is treated as a meaningful stand-in for professional CAD workflows across tools and organizations.
• Task success is emphasized over throughput, cost, recovery from failure, and the value of partially correct outputs.
• The expert reference pass rate is treated as a stable ceiling, though experts may also disagree or fail on ambiguous tasks.
• The current seven-agent comparison is implicitly informative about the trajectory of the field, despite being only a snapshot.
• The benchmark frames the problem as agent capability rather than as a labor-reorganization mechanism. That omission hides the fact that partial automation can destroy demand before full autonomy exists.

Social Function

Partial truth and transition management.

The paper is valuable because it strips away the cheap theater of successful clicks and demands a native, verifiable engineering artifact. But as a benchmark paper, it remains institutionally safe: it describes capability gaps without confronting the consequence of closing them. Its language turns a potential labor-market weapon into a neutral evaluation problem.

The 17.5% result is not reassurance. It is a map of the remaining defenses. Structural and geometric failures are exactly the barriers that delay Cognitive Automation Dominance in CAD. They are not permanent moats. They are engineering backlog.

The Verdict

CADWorld is evidence that AI has not yet achieved reliable replacement of mechanical-CAD labor. It is also evidence that the decisive battlefield has been identified: persistent, verifiable, long-horizon artifact production.

Under the Discontinuity Thesis, this is not a survival report for CAD professionals. It is an early-warning instrument. The human advantage currently consists of error recovery, structural judgment, and responsibility for valid downstream state. Once agents acquire those functions—or make them cheap enough to supervise—the wage-to-consumption circuit begins losing another skilled cognitive class. The benchmark records the gap before the breach; it does not make the wall permanent.