Cheap-model-first is the oldest cost trick in the LLM book, and I've always assumed the hard part was the router: something that looks at a task and decides whether the small model can handle it. A new AMD and NTU Singapore paper, Reducing Inference Cost of Coding Agents with Timeout Cascades, argues you can skip that part almost entirely. I think the result is real. I also think the paper's own name for it points at the wrong mechanism, and that matters if you try to copy it.
Small model first, big model on failure, no memory carried over#
The setup is clean. They ran 36 open-weight models through SWE-bench Verified on the same mini-swe-agent scaffold (250 steps, 2-hour wall-clock cap, one run per model-task pair) on MI300X GPUs, and charged each task by GPU-hours. Then they used those recorded traces to simulate two hybrid policies with GLM-4.7-Flash (30B) as the cheap model and GLM-4.7 (355B) as the fallback:
Timeout cascade: every task goes to the small model under a time limit. If it times out, produces an empty patch, breaks syntax or fails the tests, the large model starts over from the original repo state.
Difficulty router: tasks are dispatched up front by difficulty. Instead of training a classifier, they simulated one using SWE-bench's human-annotated difficulty labels, sweeping routing accuracy up to a perfect 100%.
The cold restart is a deliberately conservative choice. Nothing the small model learned about the repo gets passed to the big one, so the savings are a floor, not a ceiling.
A perfect router bought 1.5 extra points#
The headline comparison, on the 466 tasks where both models left complete trajectories:
GLM-4.7 alone: 71.2% resolved, 2585 GPU-hours
Timeout cascade at its best setting: 73.4% resolved, 2160 GPU-hours, 16.4% cheaper
Oracle difficulty router: 73.4% resolved, 2122 GPU-hours, 17.9% cheaper
Easy tasks only, cascade: 821 down to 546 GPU-hours, a 33.5% cut
So a router with access to ground-truth difficulty, which nobody has in production, beat the dumb policy by 1.5 percentage points of cost. That's the finding I'd pin to the wall of anyone about to train a task classifier for their agent platform. The authors frame it this way:
The broader insight is that for many coding workloads, timeout is not merely a failure condition; it can also be an effective routing mechanism.
The per-difficulty numbers explain why it works. The small model resolved 62% of easy tasks versus 77% for the large one, 36% versus 68% on medium, and 2% versus 36% on hard. Easy and medium make up 91% of SWE-bench Verified. When most of your workload sits where the small model is decent, a first attempt is cheap insurance, and the oracle router has very little left to optimise.
The test suite is doing the routing, not the clock#
Here's where I part ways with the framing. Look at where the best cascade setting landed: a 120-minute small-model timeout, the very top of the sweep and equal to the overall per-task cap. Shorter timeouts did worse because they killed small-model runs that would have succeeded. In other words, the savings grew as the timeout got less and less binding.
At 120 minutes the clock is barely routing anything. What decides escalation is the other trigger: the patch fails the task tests. And in this simulation those are SWE-bench's own evaluation tests, the ones that decide whether the task counts as resolved.
That's the answer key.
A production agent picking up a ticket doesn't get hidden FAIL_TO_PASS tests that tell it whether its fix is right. It gets whatever tests already exist in the repo, maybe some it wrote itself, and a model that, as several papers on this blog have shown, will cheerfully report success on work it didn't finish. If the small model's "done" is accepted on weaker evidence, two things go wrong at once: some wrong patches ship instead of escalating, and the resolve-rate parity with the big model stops being guaranteed. The paper's 73.4% versus 71.2% only holds because the escalation check is, by construction, the same check the score uses.
To be fair, the authors don't claim otherwise. They spell out the escalation conditions and they list single runs, one model pair and one benchmark as limits. The slight accuracy gain over the large model they treat cautiously too, noting it may be run-to-run variance rather than the small model's tendency to write simpler patches. But the title says timeout, and the result is really about having a trustworthy verifier at the handoff point. Those are different engineering problems, and the second one is the hard one.
Where I'd run it anyway#
None of this makes the pattern useless. It moves the work. If you run background agents (the paper points at Jules-style async workflows), here's what I'd take from it:
Build the verifier before the router. A cascade is exactly as good as the gate between stage one and stage two. Reproduction tests written from the issue before the fix, a CI run, a reviewer sub-agent with read-only access: whatever you have, the cascade inherits its false-accept rate.
Measure your task mix first. The whole effect rides on 91% of tasks being easy or medium. If your queue is mostly gnarly migrations, the small model burns time and you pay for both runs.
Don't put it in front of a human who is waiting. Average wall-clock per task went from 41.6 minutes with the large model alone to 84.5 minutes with the 120-minute cascade. That's fine for overnight tickets and unbearable in an interactive Claude Code session.
Pass the transcript on. The cold restart throws away everything the first attempt learned. Handing the large model a summary of what failed is the obvious next experiment, and the paper explicitly leaves it open.
One more caveat on the money: GPU-hours here are charged per task by wall-clock under exclusive allocation, without amortizing the up-to-30 tasks they ran in parallel. Relative savings should survive that; absolute costs on your own serving setup won't match.
Where this leaves me: convinced that an up-front difficulty classifier is the wrong first investment for a coding-agent platform, and unconvinced that a timeout is what replaces it. My bet is that the next good paper in this line keeps the cascade, swaps the hidden tests for tests the agent has to produce itself, and finds out how much of that 16.4% survives. That number is the one I'd actually budget against.