This morning's cs.CL list had a paper that answers a question I keep dodging. How long should I let a coding agent run? My sessions keep getting longer: more compactions, more tokens, more overnight runs. I had quietly assumed that more budget buys a better result. This paper measures that assumption on Claude Code, Codex, Gemini CLI and Kimi Code at up to 100M tokens per task, and it mostly breaks. Thesis: past a measurable point, a long agent session does worse than starting fresh, and you can calculate where that point is.
What it does#
When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis comes from Kaiyuan Liu, Qiuyang Mang, Luke Zettlemoyer, Alex Dimakis, Alvin Cheung and colleagues at UC Berkeley, UW, Princeton and Bespoke Labs. It starts from a real blind spot in how we evaluate agents. SWE-bench-style benchmarks give one pass/fail verdict at the end and throw away the path that led there. So the authors use open-ended tasks that score every intermediate submission: FrontierCS algorithm problems, AtCoder Heuristic Contest tasks via ALE-Bench, MLS-Bench ML research tasks, and FlashInfer-Bench GPU-kernel optimization. That is 14 problems in total. Each of four agent systems (Claude Code with Opus 4.8, Codex with GPT-5.5, Gemini CLI with Gemini 3.5 Flash, Kimi Code with K2.7) got five independent sessions per task with a target budget of 100M tokens.
Raw scores don't compare across tasks, and moving from 80% to 81% can be far harder than moving from 50% to 60%. So the paper rates progress instead of scoring it. At each token budget, the best solution so far becomes a player in a Bradley–Terry tournament (the model behind Elo ratings), which gives an Elo-per-token curve. The useful part is the baseline. The authors prove that independent sampling (run k fresh attempts, keep the best) gains exactly 400 Elo per tenfold increase in compute, whatever the score distribution. That gives every agent a line to beat. Above 400 Elo per decade, the agent's memory, feedback loops and self-revision are paying for themselves. Below it, a fresh start would have done better.
The key result#
Every agent beats the line early: adapting within a session turns tokens into Elo faster than sampling does. Then the slopes peak and fall. In the fit pooled across all benchmarks, every system drops below 400 Elo per decade by the largest budgets. The authors place the turn after the first few context-window compactions. Evolutionary harnesses (AdaEvolve, GEPA) and test-time training show the same shape. Human experts don't. Top-50 AtCoder contestants across seven 10–14-day contests improve faster the longer they work, and on AHC014 the top-10 cohort overtakes both GPT-5.6-Sol and Opus 4.8 while the agents flatten out. The authors then define the scaling inflection point: the last budget at which the curve still beats the sampling slope. For Kimi K2.7 on FrontierCS Polyomino Packing, that point is 38M tokens, so the rule says to split a 100M budget into three sessions. The split gained +264 Elo over one 100M-token session and +355 over ten 10M-token sessions, the best of the six splits tested, and the rule also held on MLS-Bench. For scale: under the Bradley–Terry model, 264 Elo means the three-session split wins about 82% of head-to-head comparisons against the single long run.
Why it matters#
Most agent tooling treats session length as good by default. Longer autonomy is a selling point, compaction is sold as the thing that lets you keep going, and "it ran for 30 hours" is a boast. This paper says that past the inflection point, keeping the session alive hurts compared with a restart. Its "sticky basin" hypothesis explains why in a way that matches what I see. Early on, the agent commits to an algorithmic idea or a code structure. More compute polishes the solution inside that basin but rarely questions the original choice, and the growing context makes that choice harder to drop. The authors show in a toy model that, as budgets grow, a session with a constant-factor larger budget beats a shorter one only half the time. That fits my experience in Claude Code: by the third compaction, the agent is usually refining the plan it chose in the first hour, not rethinking it.
Here is what I'd change as a builder. For any agent task with a continuous scorer (latency, bundle size, benchmark score, kernel speedup, a heuristic objective), log the best score so far against cumulative tokens, not just the final number. Profile that curve once on a representative task, find where it bends below the sampling slope, and cap sessions there. Spend the rest of the budget on parallel fresh sessions (subagents, separate worktrees) and keep the best result. That is depth first, then breadth, and it maps directly onto the fan-out patterns orchestrators already support. For harness designers, the lesson is that longer context isn't the goal. What matters is escaping a bad basin: making the agent reconsider, for example by starting a fresh subagent with only the spec and the scorer instead of the full history. For anyone designing evals, pass/fail leaderboards can't show this at all. Report curves.
The caveats#
Narrow task type. All 14 problems are open-ended optimization tasks with deterministic scorers. The results don't carry over directly to issue resolution or feature work, where there is no intermediate score and "keep the best of K" needs a verifier you may not have.
Thin evidence for the allocation rule. The three-session win comes from one agent on one task, plus a replication on MLS-Bench, with five sessions per cell. It's convincing enough to try, not enough to treat as a law.
You must measure the inflection point per task. A domain-level curve didn't predict the 38M-token point for Polyomino Packing, so finding it costs you the same compute you're trying to save. The authors list predicting it as open work.
Token accounting isn't dollars. Budgets count cached input tokens. The authors argue this only shifts the curves sideways, but cached and uncached tokens are priced very differently, so the split that's best for Elo may not be the cheapest. The human comparison also uses wall-clock time against tokens, so it compares curve shapes, not speed.
The takeaway#
The idea I'm keeping is that an agent's advantage over plain resampling is front-loaded and runs out, and the 400-Elo-per-decade line tells you when. Running a single marathon session is a choice, and past the inflection point it's usually the wrong one. What I'm changing: for any agent run with a real scorer, I'll plot best score against tokens before deciding how long a session should be, and I'll default to a few parallel fresh sessions instead of one that keeps compacting.