What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents

TEXT START: Agent reinforcement learning (RL) increasingly runs through full execution harnesses, and a multi-harness recipe mixes two choices: exposing the policy to several harnesses, and comparing their rewards inside one relative-advantage group.

The Dissection

This paper isolates a narrow confound in coding-agent RL. Its result is clear: the evaluation harness dominates performance, while the Cross-versus-Within grouping rule contributes no reliable portability gain. Cross learns configuration adaptation—the harness leaves a fingerprint in the policy—not general coding capability.

The useful finding is methodological: benchmark scores are heavily harness-conditioned. The dangerous overreading is that weak portability implies weak automation. It does not.

The Core Fallacy

The paper treats portability across harnesses as the relevant unit of generality. Under Discontinuity Thesis mechanics, the relevant unit is economically substitutable output inside a deployed workflow. An agent need not function everywhere. It needs to be cheap and effective inside a workflow whose harness can be tailored by its owner.

Harness dependence is therefore an integration cost and a temporary moat, not protection for human labor. The paper does not test P1, P2, or P3: it measures benchmark transfer, not durable cost superiority, the impossibility of preserving human-only cognitive domains, or the collapse of mass productive participation.

Hidden Assumptions

• SWE-bench solve rate represents economically meaningful coding productivity.
• Four source harnesses and one minimal held-out harness represent the deployment frontier.
• A non-significant Cross-minus-Within result reflects no meaningful effect rather than limited power; the confidence intervals still permit a positive portability gain.
• Harness adaptation is separate from capability rather than one form of capability valuable to a deployer.
• Portability is necessary for labor substitution.
• Benchmark action distributions and oracle scores capture maintenance, security, integration, latency, and cost.

Social Function

Partial truth, prestige signaling, and transition management. It punctures inflated claims that multi-harness RL automatically produces portable agency and correctly demands unseen-harness testing. But it confines the crisis to benchmark hygiene. Read lazily, it becomes an ideological anesthetic: agents are not universal, therefore humans are safe. That inference is unsupported.

The Verdict

This is a competent autopsy of a benchmark confound, not a reprieve for coding labor. It shows that Cross-harness credit assignment does not create portable capability. It does not show that specialized, proprietary agents cannot replace human coders inside controlled environments.

Under the DT lens, the likely outcome is not one universal coding agent but fleets of harness-specific systems owned by firms that control repositories, compute, evaluation, and deployment infrastructure. Harness dependence delays substitution and raises the value of integration and maintenance. It does not reverse the machine. The paper measures a lag in automation, not its defeat.