Tuesday's cs.SE list had a benchmark I'd been waiting for someone to build: a test of whether memory systems actually help coding agents on real repositories. Every agent setup I use now has some form of memory, whether that's CLAUDE.md, auto-memory files, or a vector store behind a plugin, and almost nobody measures whether it earns its context. This paper does, and the answer is uncomfortable. My thesis: memory for coding agents is a writing problem before it is a retrieval problem.
What it does#
VibeMemBench (Fan, Shi, Li et al., Shenzhen Institutes of Advanced Technology and Alibaba) builds 111 coding targets across 90 repositories from SWE-rebench V2. Each target is a repo snapshot, an instruction, and a hidden executable test, and each comes with earlier history from the same repo: 3,634 completed agent trajectories on related issues in total. The construction pipeline screens for history that touched the same files or directories as the target, runs it with an agent, and distills each run into a short "verified experience" record with four fields: bug class, root cause, fix pattern, and a reusable lesson. The median record is 10 lines. The leakage controls look careful. The target's gold patch is only used in offline screening, and the test patch never leaves the evaluator.
What I like is the split into two conditions. First, inject the hand-verified record directly, which sets a ceiling on what the history is worth. Second, let four off-the-shelf memory systems (Mem0, SimpleMem, MemoryOS, A-MEM) ingest the same history themselves and retrieve their top record for the target. Everything runs on the same MiniSWEAgent harness with a fixed context budget and four seeds per condition, on solvers including deepseek-v4-pro, glm-5, glm-5.2, kimi-k2.7-code and qwen3.8-max. Most memory papers stop at "we added memory and the number went up." This one measures what the history is worth, then how much of that the memory layer actually delivers.
The key result#
The history is worth something, but not much. Injecting the verified record adds between 0 and 4.5 points of resolution depending on the solver (kimi-k2.7-code +4.5, glm-5.2 +3.6, glm-5 +2.4, qwen3.8-max +1.1, deepseek-v4-pro +0.0), and it cuts agent steps on all five (glm-5 drops from 77.1 to 66.2). The memory systems deliver none of that. Eleven of twelve solver–system pairings fail to beat the matched memory-off baseline, and eleven of the twelve point estimates are negative, reaching −5.5 points for Mem0 on glm-5, the one interval that excludes zero. When the authors trace 231 failing retrieval pairings, only 1.3% are ranking misses. 16.0% are coverage misses, 13.4% are the solver ignoring a good record, and 69.3% are "form degradation": the right record came back, but it was written in a way that buried or mangled the fix. One example stuck with me. On aiohttp-7907, A-MEM injects a 334-line transcript from a different issue, which sends the agent to the wrong file and takes it from 4/4 seeds resolved to 0/4. A one-line Mem0 record on the same target keeps it at 4/4.
Why it matters#
The failure breakdown matches things I see in real setups. Instruction pollution (64 of the 160 form failures) is a memory system storing the old task's instructions verbatim, so the agent reads someone else's issue as part of its own. Overgeneralized fixes (29) come from extraction prompts that flatten a specific, file-anchored fix into something like "handle errors carefully." Tool and shell noise (16) is a chunker keeping raw command output. The ablation I'd tape to the wall: deleting the flagged instruction lines recovers 0.60 seeds per pairing, and deleting the same number of random lines recovers 0.48. Most of the harm comes from volume, not from any particular bad sentence. So if you maintain CLAUDE.md files, auto-memory, or a lessons file that sub-agents append to after a task, the rule is simple. Store the fix, anchored to a file or identifier, in a few lines. Don't store the transcript, the task description, or tool output. The records that worked had a median of 10 lines. The system records averaged 65.
The second finding changes how I'd wire memory into a harness. The same record helps a weaker solver and hurts a stronger one. When a solver resolves a target 0/4 without memory, anchored records help in 55.2% of pairings with no observed losses. When it already resolves 4/4, even the hand-verified experience lowers resolution in 25.7% of pairings, and 42% of all retrieval pairings sit at that ceiling. A rule that looks only at solver headroom predicts the sign of the effect better than one that looks only at the record (56.5% vs 52.4%, not a huge margin, but the direction is clear). In practice that argues against loading memory into every session at startup. Put it behind a trigger instead: after a failing test run, after the agent has circled the same files a few times, or after a cheap first attempt fails. In a sub-agent architecture, the orchestrator can hold memory and hand a record to a worker only when that worker gets stuck. And if you're evaluating a memory feature, split results by baseline difficulty, because an average can hide gains on hard tasks and losses on easy ones.
The caveats#
No Claude or GPT models in the solver pool, and everything runs on MiniSWEAgent. A frontier model in a production harness might shrug off a noisy record, or it might sit at the ceiling more often, where records can only hurt.
Targets were kept only if the verified record helped deepseek-v4-flash, so the benchmark is enriched for cases where history should help. That makes the memory systems' failure more striking, but the oracle gains aren't a prevalence estimate, and every per-solver oracle interval crosses zero. "A few points, at best" is the honest summary.
Retrieval happens once, before the run. Closed-loop memory calls during solving, which is where most agent memory tools are heading, aren't tested.
Token cost and latency of the memory systems weren't measured.
The takeaway#
What I'm filing away: the bottleneck in agent memory isn't finding the right past experience. It's writing that experience down in a form that doesn't drown the current task, and knowing when to leave it out. Short, anchored, fix-first records, injected when the agent is struggling rather than every time. After reading this, I'm cutting my project memory files down to fix-plus-anchor entries and moving the "lessons learned" block out of the session preamble into a hook that only fires after a failing test run.