terms.txt: A Consent and Compensation Protocol for Agentic Web Access

TEXT START: The open web ran on an unwritten bargain: sites admitted crawlers, and search engines sent visitors back.

The Dissection

This paper tries to convert machine access from an informal, unenforceable courtesy into an authenticated commercial transaction. Its terms file, signed intent, delegation tokens, HTTP 402 negotiation, and receipts create a control layer for identifying agents, declaring purpose, charging access, and auditing compliance.

It correctly identifies the immediate fracture: AI systems consume enormous volumes of web material while returning little traffic or value to publishers. But it frames a structural power seizure as a missing protocol feature. The proposal builds a tollbooth around the corpse of the old web bargain.

The Core Fallacy

The central error is confusing chargeability with bargaining power, and bargaining power with survival.

A payment rail may let an origin demand compensation. It does not ensure that the dominant AI firms will accept the price, that small publishers can enforce it, or that captured revenue will restore human productive participation. Under P1, machine systems continue replacing cognitive labor. Under P2, institutions cannot preserve stable human-only economic territory merely by adding a standard. Under P3, money routed to sites does not recreate the mass employment-to-wage-to-consumption circuit.

The protocol can monetize extraction. It cannot prevent extraction, reverse automation, or stop ownership of the web’s productive value from concentrating in a few AI platforms and infrastructure providers.

Hidden Assumptions

• Agents will remain dependent on live origin access instead of using caches, licensed corpora, private indexes, user-mediated retrieval, or substitute data.
• AI platforms will pay rather than bypass, substitute, consolidate suppliers, or impose take-it-or-leave-it prices.
• Origins can reliably authenticate hostile or distributed clients and enforce terms across jurisdictions.
• Publishers can coordinate on prices and terms instead of entering a race to the bottom.
• Signed identity and stated purpose will correspond to actual downstream use.
• Per-request compensation will track the economic value extracted from content.
• Revenue captured by websites will reach creators, workers, or local economies rather than merely strengthening platform owners.
• Major browsers, CDNs, hosting providers, and AI firms will adopt a common standard instead of preserving proprietary control points.

The claimed 0.20–0.65 ms overhead is operational trivia beside these assumptions. The decisive bottleneck is not latency. It is who controls access, substitutes, capital, and enforcement.

Social Function

Primary classification: partial truth and transition management. Secondary classification: elite self-exoneration and ideological anesthetic.

The paper is not pure copium. It offers a potentially useful mechanism for publishers seeking compensation and auditability during the transition. Its local diagnosis is real: robots.txt is too weak to express identity, intent, price, or enforceable terms.

Its ideological function is larger than its technical function. It suggests that the crisis can be repaired by making machine access legible and billable. That relocates the conflict from ownership and displacement to protocol design. Content owners get a new rent-collection mechanism; AI firms get a standardized market for buying access; everyone can pretend the web’s social contract has been upgraded rather than terminated.

The Verdict

This is a competent receipt printer for the funeral, not a resurrection machine.

At best, it creates a temporary moat for publishers and helps convert free extraction into metered AI rent. At worst, it becomes a compliance layer that dominant AI firms can afford, negotiate down, or route around while smaller sites are priced out and consolidated.

The proposal may preserve revenue for selected owners. It does not defeat cognitive automation, solve coordination failure, or restore productive participation for the displaced majority. It is transition infrastructure for a more explicitly machine-owned web—not a defense of the post-WWII economic order.