The largest prompt in this agent's entire evaluation used 5.75% of a 400k-token window. About 23k tokens. And yet the model was working from broken data, because the harness around it had cut a third of what its tools returned. I keep telling people to look at the window size last. This is the clearest evidence I have seen for why.
The paper is Context Is a Pipeline, Not a Window: Information Attrition in LLM Agent Harnesses, from Sehgal, Panahandeh and Ezzati-Jivan. Their agent finds root causes of injected faults by reading Linux kernel traces. The field evaluation covers 720 runs and 11,189 tool results, and the headline is blunt: the context window was almost empty, and information was still being lost.
The cut is deterministic, and it wrecks the format#
The shipped harness truncated tool output to a character budget. That sounds harmless. In practice it hit 3,827 of 11,189 tool results (about 34%), and the authors report that every cut result became broken JSON. The model did not see a shorter answer. It saw a malformed one.
The loss was also uneven in a way you would never spot from logs. Inside the same tool results, one field reached the model in 4.7% of 742 calls while another reached it in 99.3%. Whichever field sat late in the payload simply stopped existing.
50% of process-listing calls broke, 43% of analysis outputs, 1% of small summaries
Only 2.5% of all calls were affected in the replay set, so averages look fine
In one production trace, the cut delivered 30 of 118 early-list records and 0 of 128 late-list records; the listing was 33,370 characters, 5.6× the budget
That last number is the one I would pin to the wall. Records at the end of a list are not degraded. They are gone.
Fixing it moved accuracy from zero to half#
In the production replay, the repaired harness named the correct service in 33 of 66 runs. With the defect in place it was correct in 0 of 55 runs, with every payload broken. The authors are careful that this link is correlational and noisier than their controlled experiment, and I agree with that framing. But 0 of 55 against 33 of 66 is not a rounding error.
The controlled experiment is more interesting, and more humbling. On spawned-culprit faults the broken harness was still right 15 of 15 times, the same as the repaired one at 14 of 15. What changed was effort:
Repaired: median 53 calls, 18 computations
Defect: median 68 calls, 32 computations
Defect reversed: median 76 calls, 36 computations
Calls rose 28%, then 43%, and computations roughly doubled. The agent routed around the damage. Their summary: "The agent still finds the right answer when it has another way to reach it."
This is the part that should worry anyone running evals on pass rate alone. A silently lossy harness can show up as a cost bill rather than a failure. Accuracy only collapses for a quiet culprit with no second route to the answer, which is exactly the case you least want to discover in production.
Five stages, and the window is only one of them#
The framing is what I would steal. The authors model context as a pipeline: produce, deliver, act, retain, transfer, persist. They measure failures at four of those stages and treat the last, persistence, as a deliberate design choice. Most of us only tune the middle. We pick a bigger window or a better compaction prompt, and never ask what survived the trip from tool to prompt.
Another number from the paper supports the same point from the other direction: in two production coding agents, waiting on the model took a median 84% of wall-clock time across 10 uncapped runs. So extra calls caused by lost information are not free. Each one is a model round trip, and that is where your latency and your bill live.
Your framework may do this by default#
The authors surveyed tool-output cutting in seven agent frameworks. Two cut by position: langchain-core keeps the first 500 characters, and smolagents deletes the middle of observations over 20k characters. Three do not cut output, one cuts only tracing data, and one structure-aware cutter exists but is not the default.
I would not extrapolate beyond that. This is one harness, one domain (kernel traces) and one model. The authors did not measure a conventional build-test-fix coding agent end to end, and they wrote the defect themselves, then reintroduced it under registered predictions rather than finding it in the wild. The experiment is also small: repeats of the same recordings, not independent fault instances. They even caught their own scorer inflating one arm from 2/15 to 14/15 and replaced it, which I trust more than a clean table.
What I am checking in my own harnesses#
Claude Code and sub-agent setups all have some tool-result budget somewhere. The paper makes me want three things:
A log line whenever any tool result is truncated, with the original and delivered sizes
A rule that truncation never produces invalid structure: cut whole rows or records, or return a summary plus a pointer, not a sliced string
An eval arm that reports calls per solved task next to the pass rate, so lost information shows up as effort
My prediction: a lot of agent flakiness we blame on the model is attrition between the tool and the prompt. If your agent seems oddly persistent on some tasks and oddly blind on others, check what it was actually handed before you swap models.