When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

TEXT START: Long-term memory for LLM agents is evaluated today by conversational recall benchmarks (LoCoMo, LongMemEval), which measure question answering over dialogue history, not whether remembered facts change what a tool-using agent does.

The Dissection

This paper converts “memory” from a demo feature into an instrumented production variable: task success minus token, dollar, corruption, and retrieval costs. MERIT exposes that implementation—not memory in the abstract—determines whether agents execute correctly. Its strongest finding is operational: update-on-write stores outperform naïve embedding retrieval, while full replay is economically wasteful.

The Core Fallacy

The paper’s blind spot is scale confusion. It treats memory as an engineering bottleneck inside the agent stack, not as another mechanism accelerating the replacement of human cognitive labor. Better memory does not preserve productive participation. It raises autonomous-agent reliability, lowers the cost of execution, and strengthens the owners of AI capital. The benchmark measures whether agents work better; it says nothing about whether humans remain economically necessary.

Hidden Assumptions

• The three task domains represent deployment reality.
• Automated leak checks and success scores capture genuine task dependence and business value.
• Token and dollar metering reflects durable operating economics.
• Tested models, seeds, and memory implementations generalize under larger scale, conflicting facts, adversarial inputs, and long deployment horizons.
• LLM summarization will remain reliable as memory volume and consequence increase.
• The benchmark’s “best condition” remains best as models, prices, and tool environments change.
• Correct retrieval is a meaningful proxy for safe, economically valuable action.

Social Function

Classification: partial truth, transition management, and prestige signaling. It is not empty copium. It correctly kills the lazy assumption that any retrieval layer produces useful memory. But socially, it functions as a maintenance manual for the machinery of substitution: improve state persistence, reduce waste, and make autonomous agents cheaper to operate.

The Verdict

MERIT is a useful autopsy of memory systems and a poor defense of human relevance. Its result is structurally grim: memory is not a durable moat; it is a control layer whose best implementations increase the marginal utility of autonomous labor per dollar. That creates temporary engineering niches while pushing the system further toward P1 and P3. The paper solves a bottleneck in the machine. It does not alter the machine’s destination.