If your agent harness has a failure alarm, I'd bet you haven't checked what it is alarming on. Shi and Assunção did that check in Hard Issue, Weak Model, or Bad Run?, and the answer is uncomfortable: a baseline that knows only how hard the issue is reaches 0.924 AUROC at predicting failure. The trajectory-based predictors reach 0.62 to 0.63.
That gap is the whole paper. Pooled AUROC looks great for any predictor that picks up difficulty, and it says nothing about whether a given run went wrong.
Almost every pair being scored is from different issues#
The setup is public mini-swe-agent trajectories: 3,492 runs from seven models, 994 of them failures, one attempt per model per issue. The predictors are a TF-IDF logistic regression and a LoRA-tuned Qwen3-0.6B, reading the first 10, 20 or 40 steps. They never see the issue text or the model name, which sounds like a fair test of whether the trajectory carries signal.
Here is the catch. 99.66% of the pairs that pooled AUROC compares are from different issues. So the metric mostly rewards telling an easy issue from a hard one. Split it into within-issue pairs, where the same issue was solved by one run and failed by another, and the picture changes:
Issue difficulty alone: 0.924 pooled AUROC.
Model identity alone: 0.72 within-issue (0.82 with 23 models).
Trajectory predictors: 0.60 to 0.61 within-issue, so they lose to simply knowing which model ran.
On the cases where a normally failing model won (upsets), predictors scored 0.42, and all ten TF-IDF configurations fell below 0.5.
Add the trajectory score on top of difficulty and model identity and pooled AUROC moves by -0.001, within-issue by -0.011 (95% CI -0.024 to 0.002). The authors put it bluntly:
The trajectory adds nothing measurable beyond both.
Model switching is a story the data doesn't support yet#
The tempting next step is to use the alarm: if the run looks doomed, hand the issue to a different model. This is the pattern behind a lot of routing and escalation logic, and the paper tests it.
The best single model solves 76.8% of issues and any of the seven solves 86.8%, a 10-point oracle gap. That reads like room for specialization. But the observed gap and upset rate sit inside what a no-specialization model (Rasch and 2PL fits) produces by chance. The practical numbers follow:
Per-issue choice from past runs: +0.09 points (CI -0.36 to 0.63).
A router for new issues: +0.41 points (CI -1.83 to 2.85).
Escalating failures to a second model: +6.91 points, but with a CI of -4.67 to 9.15, and the authors credit the extra attempt, not better matching.
That last one matters for anyone running retries. A second attempt helps because it is a second attempt. Run-to-run variation explains 42% of outcome variance here, issue difficulty 53%, and specialization only 4.6%. A fresh sample from the same model competes with a router for the same gain, at equal budget.
A high score means a hard issue, mostly#
Predictor scores were 64 to 82% between-issue variance. So when your alarm fires at step 20, the likeliest reading is that the issue is hard or the model is weak, not that this particular run took a bad turn you could fix by restarting. Only about 15 of 522 issues showed recurring upsets in the repeated-run pilot (two Llama models, at least four runs per issue).
That doesn't mean bad runs are rare, since run variance is the 42%. It means the alarms can't tell you which kind of failure you are looking at, and the remedy differs: a bad run wants a retry, a hard issue wants a better model or a human.
Why the pooled number keeps fooling people#
This is an old trap in a new place. If you evaluate a classifier on a mix of easy and hard cases, it can score well by sorting the cases and never looking at what happened inside them. Here the trajectory features that survive in a model like Qwen3-0.6B are mostly proxies: long runs, repeated edits and flailing are things hard issues cause, whoever is driving. The predictor learns the weather, not the run.
The authors also built a reference for what chance alone would produce, using Rasch and 2PL item-response fits. That is the part I found most convincing. It turns a vague feeling that models specialize into a testable claim, and the claim didn't clear the bar.
What I'd do differently in a Claude Code style harness#
If you orchestrate sub-agents or retries, log the issue and the model with every outcome, then compute your alarm's accuracy per task, not across the whole pool. If a cheap difficulty prior (how often other models failed this task) beats your signal, your signal is a difficulty meter. And when you test a router, hold out runs, not just issues, or you will credit the router with what resampling would have given you.
What would change my mind#
The evidence is thinner than the headline. It is one benchmark and one scaffold, mostly reusing released labels. Only trajectory-based predictors were tested, so a predictor that sees the issue text, test output or diff might do better. The repeated-run pilot uses older models with a router fed shallow issue features, and failing to reject no-specialization is not proof that specialization is absent.
Still, the method is the part I'd steal. Before trusting any failure alarm in a harness, report within-issue AUROC, compare against a run-blind baseline of difficulty plus model, and judge the action it triggers at an equal attempt budget. My guess is that most home-built early-stopping heuristics fail that test, and that the honest ones will end up as cheap budget caps rather than predictors.