Scrolling the morning arxiv list, most of this week's agentic-coding papers were the usual crowd: another benchmark, another safety probe, another “resolve rate hides X” critique. Then PerfAgent stopped me, because it measures something almost nobody in this corner of the field does. Not whether an agent can fix a bug, but whether it can make real code faster without breaking it. And underneath the perf story is a general lesson about agent design. My thesis for this post: the highest-leverage knob in an optimization agent is the quality of the feedback signal, not the amount of compute you throw at it.
What it does#
PerfAgent (Deng, Liu, Lipka, Ma, Chen, Kaler, Ganhotra) tackles repository-level code optimization — preserve behavior, improve runtime — which is a strictly harder feedback problem than SWE-bench-style issue resolution. There's no failing test that flips green; “better” is a continuous speedup ratio and “correct” is a hard constraint you can silently violate. The authors name three ways agents flunk this: they miss the real bottleneck when it hides behind an abstraction layer or a native extension, they quit after the first shallow speedup, and they under-test and ship a silent correctness regression.
The answer is a profiler-in-the-loop controller. PerfAgent runs py-spy — a sampling profiler that sees into C/C++/Cython/Rust frames, not just Python — filters the raw trace into hotspots (location, call context, self-time, total-time, external-library flag), and then, the nice touch, uses a separate LLM call to condense the profile into a natural-language summary so it doesn't blow up the context window. The agent edits, submits a patch, and a controller rebuilds, runs only the affected tests via pytest-testmon (cutting the test set by 66–99%), and if it passes, re-profiles and hands back the measured speedup plus a fresh hotspot summary. It loops up to five times and keeps the fastest correct patch across iterations, not the last one the agent happened to submit. Unlike prior profiler-in-the-loop work aimed at GPU kernels or model pruning, this is real repositories — NumPy, pandas, pydantic — held to a human-expert performance bar.
The key result#
On SWE-fficiency-Lite the jump is the one to quote: from 26% to 74% expert-matching patches — patches that reach at least 0.95 of the human expert's speedup and pass correctness. On the harder GSO benchmark (102 tasks across ten repos, 59% of which touch non-Python code) it roughly doubles, 19.6% to 39.2%. But the number I actually care about is the cost comparison. PerfAgent beats an oracle best-of-five OpenHands baseline — a baseline that gets to run five independent times and keep its single best result after the fact — while costing $2.88 per task against $11.01 on GSO ($4.25 vs $9.91 on SWE-fficiency-Lite). That's the whole argument in one row: the gains come from better feedback, not from sampling harder. Give it more budget and it keeps climbing modestly (75% → 79% on SWE-fficiency-Lite at $10/task), but the point stands that a single well-fed loop out-performs a five-shot lottery at a fraction of the spend.
Why it matters#
The feedback-signal lesson generalizes well beyond performance. It's the same conclusion the verified-crypto and dockerless-verifier work keep landing on: what an agent can be trusted to establish is bounded by the strength of its feedback. Optimization just makes it vivid, because the naive signal — wall-clock timing — is noisy and points nowhere, while a profiler tells the agent where the time actually goes, including in native frames it would otherwise never see. If you're building any agent whose success is continuous rather than binary (perf, cost, quality, relevance, retrieval), the design question isn't “which model” or “how many samples” — it's “what's the richest, most actionable signal I can put in the loop, and can I afford to recompute it every turn?”
The ablation is where the concrete builder advice lives. The profiler is load-bearing: loop-plus-profiler hits 36.3% on GSO, but loop-plus-tests alone drops to 24.5% — below the 33.3% of a bare loop. Read that twice. Naive test feedback without the profiler made the search worse, presumably by letting the agent chase passing-but-pointless edits with no sense of where the cost is. Two moves fall out. First, don't let your optimization or refactor agent stop at the first green patch; wrap it in an objective-driven controller that re-measures and retains best-of-N-iterations. Second, make verification cheap enough to run every single turn — selective test execution (testmon-style) is exactly what makes profile→edit→verify→reprofile viable instead of a per-iteration tax you're tempted to skip.
The caveats#
Gameable half. SWE-fficiency-Lite has no hidden tests, so speedups can be faked; the authors lean on a reward-hacking detector (LLM-judge plus stack introspection) that they admit can miss subtle exploits or over-flag legitimate wins. GSO's hidden performance tests are the more trustworthy half.
Blind spot in the verifier. pytest-testmon only tracks Python coverage, so a regression inside a native extension can slip between iterations — ironic, given native frames are exactly where the speedups come from.
Amplifier, not leveler. The scaffold mostly makes a strong model stronger. GPT-5.1 nearly triples on SWE-fficiency-Lite; open-source Kimi-K2 moves 9.8% → 10.8% on GSO (about two tasks, inside the noise).
Narrow evidence base. Two benchmarks, ~12 Python-centric repos, one base harness (Mini-SWE-Agent), single runs. Harness- and language-agnostic transfer is asserted, not shown.
The takeaway#
What I'm filing away: performance work is the cleanest case study for a principle I keep relearning — spend your budget on the feedback signal before you spend it on compute. A profiler beats a stopwatch, best-of-N-iterations beats best-of-N-samples, and a cheap re-verification you run every turn beats an expensive one you run once. The next time I reach for best-of-5 to paper over a weak signal, I'm going to ask first whether I can just make the signal richer instead.