Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

TEXT START: Graphical user interfaces have emerged as an important environment for evaluating autonomous AI agents on multimodal interactive tasks.

THE DISSECTION

The paper expands the testing surface for mobile agents and frames their failures as an engineering problem: better benchmarks, better context retention, and better state tracking. Its useful finding is blunt: current agents become unreliable when tasks become long, stateful, and realistic.

But GMA measures execution reliability, not economic displacement. It tests whether agents can complete selected workflows, not whether AI is becoming cheaper and more capable than human cognitive labor across the economy.

THE CORE FALLACY

The implicit error is treating present-day brittleness as evidence of durable human advantage. Under the Discontinuity Thesis, imperfect automation is still automation. The relevant question is not whether agents can flawlessly perform every workflow today. It is whether their cost and performance are improving toward durable superiority.

GMA’s results are therefore lag evidence, not a rebuttal to P1. Harness improvements—context retention, explicit state tracking, and workflow coordination—are not permanent moats. They are engineering instructions for removing the friction that currently protects human labor.

HIDDEN ASSUMPTIONS

• Seven open-source applications and 300 tasks adequately represent real mobile work.
• Eight frontier models reveal the field’s durable ceiling rather than its current position.
• High failure rates will remain economically unacceptable as systems improve.
• Human involvement will continue to be required at the same scale when agents can route, retry, verify, or escalate failures cheaply.
• Better harnesses will merely assist human workers rather than progressively replace them.
• UI interaction is the main bottleneck, while ownership, deployment cost, infrastructure, and labor-market coordination remain outside the analysis.

The abstract also provides no evidence about P2 or P3: whether institutions can preserve human-only economic domains, or whether most people retain access to economically necessary labor.

SOCIAL FUNCTION

Primary classification: partial truth. Secondary classification: prestige signaling and transition management.

The paper accurately documents current agent weakness. But its benchmark framing channels attention toward scores, harnesses, and workflow reliability—the manageable surface of the problem. That turns a potential labor-market rupture into a familiar software optimization contest. The danger is not that the benchmark is false. The danger is that it is true in a way that can be mistaken for safety.

THE VERDICT

GMA is evidence that mobile agents are not yet reliable enough for unrestricted substitution. It is not evidence that human cognitive labor has a lasting moat. Complexity, statefulness, and interface friction are lag defenses; harness design is already aimed at dismantling them.

Under DT logic, this benchmark is an early warning instrument misread as a safety certificate. It records the temporary weakness of the servitor layer while helping engineers remove that weakness. The abstract establishes present immaturity—not human economic durability.