← all writing
03 · 20 Nov 2026 · 6 MIN READ

A timeout rule nearly matched a perfect router, but the hidden tests did the routing

Cheap-model-first is the oldest cost trick in the LLM book, and I've always assumed the hard part was the router: something that looks at a task and decides whether the small model can handle it. A new AMD and NTU Singapore paper, Reducing Inference Cost of Coding Agents with Timeout Cascades, argues you can skip that part almost entirely. I think the result is real. I also think the paper's own name for it points at the wrong mechanism, and that matters if you try to copy it.

Small model first, big model on failure, no memory carried over

The setup is clean. They ran 36 open-weight models through SWE-bench Verified on the same mini-swe-agent scaffold (250 steps, 2-hour wall-clock cap, one run per model-task pair) on MI300X GPUs, and charged each task by GPU-hours. Then they used those recorded traces to simulate two hybrid policies with GLM-4.7-Flash (30B) as the cheap model and GLM-4.7 (355B) as the fallback:

The cold restart is a deliberately conservative choice. Nothing the small model learned about the repo gets passed to the big one, so the savings are a floor, not a ceiling.

A perfect router bought 1.5 extra points

The headline comparison, on the 466 tasks where both models left complete trajectories:

So a router with access to ground-truth difficulty, which nobody has in production, beat the dumb policy by 1.5 percentage points of cost. That's the finding I'd pin to the wall of anyone about to train a task classifier for their agent platform. The authors frame it this way:

The broader insight is that for many coding workloads, timeout is not merely a failure condition; it can also be an effective routing mechanism.

The per-difficulty numbers explain why it works. The small model resolved 62% of easy tasks versus 77% for the large one, 36% versus 68% on medium, and 2% versus 36% on hard. Easy and medium make up 91% of SWE-bench Verified. When most of your workload sits where the small model is decent, a first attempt is cheap insurance, and the oracle router has very little left to optimise.

The test suite is doing the routing, not the clock

Here's where I part ways with the framing. Look at where the best cascade setting landed: a 120-minute small-model timeout, the very top of the sweep and equal to the overall per-task cap. Shorter timeouts did worse because they killed small-model runs that would have succeeded. In other words, the savings grew as the timeout got less and less binding.

At 120 minutes the clock is barely routing anything. What decides escalation is the other trigger: the patch fails the task tests. And in this simulation those are SWE-bench's own evaluation tests, the ones that decide whether the task counts as resolved.

That's the answer key.

A production agent picking up a ticket doesn't get hidden FAIL_TO_PASS tests that tell it whether its fix is right. It gets whatever tests already exist in the repo, maybe some it wrote itself, and a model that, as several papers on this blog have shown, will cheerfully report success on work it didn't finish. If the small model's "done" is accepted on weaker evidence, two things go wrong at once: some wrong patches ship instead of escalating, and the resolve-rate parity with the big model stops being guaranteed. The paper's 73.4% versus 71.2% only holds because the escalation check is, by construction, the same check the score uses.

To be fair, the authors don't claim otherwise. They spell out the escalation conditions and they list single runs, one model pair and one benchmark as limits. The slight accuracy gain over the large model they treat cautiously too, noting it may be run-to-run variance rather than the small model's tendency to write simpler patches. But the title says timeout, and the result is really about having a trustworthy verifier at the handoff point. Those are different engineering problems, and the second one is the hard one.

Where I'd run it anyway

None of this makes the pattern useless. It moves the work. If you run background agents (the paper points at Jules-style async workflows), here's what I'd take from it:

One more caveat on the money: GPU-hours here are charged per task by wall-clock under exclusive allocation, without amortizing the up-to-30 tasks they ran in parallel. Relative savings should survive that; absolute costs on your own serving setup won't match.

Where this leaves me: convinced that an up-front difficulty classifier is the wrong first investment for a coding-agent platform, and unconvinced that a timeout is what replaces it. My bet is that the next good paper in this line keeps the cascade, swaps the hidden tests for tests the agent has to produce itself, and finds out how much of that 16.4% survives. That number is the one I'd actually budget against.


Working on something similar?

Say hello — I read every email.