LRU is harder to beat than the KV-cache papers suggest

TEXT START: I replayed 68,266 requests from 393 real Claude Code sessions and 23,608 Mooncake requests through a prefix-cache simulator, tried to beat the production baseline three different ways, and failed.

The Dissection

This is a serious falsification report of a narrow optimization thesis: that liveness-aware eviction can beat radix-leaf LRU in agentic KV caching. Its real contribution is regime separation. In the supplied experiments, tight two-second tool loops generated more recomputation than long idle gaps; capacity pressure dominated the five-minute TTL; and three policy improvements made the result worse. The harness audit also exposed why an unpinned simulator can accidentally make LRU look superior.

The strongest defensible conclusion is conditional: on these capacity-bound traces, with these cache sizes and workload assumptions, LRU is a formidable baseline and liveness prediction targets the wrong waste term.

The Core Fallacy

Technically, the headline outruns the evidence. The study uses 393 sessions, a single dense Mooncake hour, synthesized AgentX arrival times, session-local hashes that erase cross-session sharing, and a 40-session ablation. It does not establish that LRU beats liveness-aware policies in TTL-bound provider caches, human-in-the-loop workflows, or workloads with materially longer idle gaps. The unresolved 4–6 percentage-point validation offset further limits how broadly the measurements can be trusted.

The deeper Discontinuity Thesis fallacy is treating cache policy as the decisive question. It is not. Whether LRU, compression, tiering, admission control, or scheduling wins, the successful optimization lowers the cost of machine cognition. This work does not resist P1; it helps operationalize it. The contest is over which mechanism makes cognitive automation cheaper, denser, and more reliable. LRU winning a local benchmark is not human economic survival. It is the replacement engine running efficiently.

Hidden Assumptions

• The sampled traces approximate future production workloads closely enough to guide general policy.
• Synthetic session arrivals do not materially distort contention, concurrency, or cache occupancy.
• Effective recompute cost is the relevant objective, rather than latency, throughput, energy, SLOs, or implementation complexity.
• GPU execution, memory bandwidth, compression overhead, and tiering behavior do not reverse the simulator’s ranking.
• The capacity-bound regime is the strategically important regime, despite the text conceding that provider caches can be TTL-bound.
• The absence of cross-session prefix sharing is conservative rather than structurally decisive.
• A strong LRU baseline will reduce serving costs without triggering enough workload expansion to consume the savings.
• Improving AI serving infrastructure is economically neutral. Under the DT lens, it is not neutral; it increases the reach of automated cognitive labor.

Social Function

Partial truth, prestige signaling, and transition management. This is not crude copium. The author reports failed hypotheses, unresolved discrepancies, limitations, and a simulator bug that damaged the preferred baseline comparison. That honesty gives the work credibility.

Its social function is still larger than its technical claim. It converts the expansion of machine cognition into an engineering problem of cache efficiency, validates a familiar production default, and directs attention toward making AI inference cheaper. For serving engineers, it is useful. For the labor system, it is another maintenance memo for the machine that is removing the need for labor.

The Verdict

The article successfully kills one fashionable idea in one measured regime: liveness-aware eviction did not beat LRU under capacity pressure because the dominant waste came from near-contiguous tool loops, not expired sessions. It does not kill the liveness literature, and it does not generalize across TTL-bound or radically different workloads.

Under the Discontinuity Thesis, this is not a moat. It is a tuning note. The cache may remain LRU; the economic consequence remains the same: more productive cognition moves from human workers into owned computational capital, while the majority are left outside the circuit that once converted labor into wages and consumption.