Scrolling this morning's arxiv list, one title stopped me because it argues against something I'd quietly treated as load-bearing: that verifying a coding agent's patch means running its tests. "Dockerless" claims you can judge patch correctness without executing anything — just let a sub-agent read the repository. My reflex was skepticism, so I opened it to find the flaw. Then I read the numbers. My thesis after doing so: the expensive, execution-based half of your agent's verification loop is far more replaceable than the field has been assuming — and that quietly reshapes how you build a training pipeline.
What it does#
Program verifiers are one of those pieces of infrastructure that stay invisible until you try to train an agent. They decide which rollouts are good enough to keep for supervised fine-tuning, and they hand out the reward signal during RL. The standard way to verify a patch is to run the repository's unit tests — which means building a per-repository Docker image, resolving pinned dependencies, and eating all the environment setup cost that entails.
Dockerless (Zeng et al., June 2026) throws execution out entirely. Instead of running tests, it treats verification as an agentic reading task. From the issue and a reference patch it generates two to four verification questions — where the fix should apply, what the code is supposed to do, how you'd confirm it, and what else might break. Parallel sub-agents then explore the repo with read-only shell tools (find, grep, rg), gather evidence-backed answers, and a judgment stage aggregates those question–answer pairs into a binary verdict, softened into a continuous score via a softmax over logits. No container, no test run. What sets it apart from earlier LLM-as-judge verifiers is that the verdict is grounded in retrieved repository evidence, not the model's prior about whether a diff merely looks plausible.
The key result#
The headline is that a verifier which never executes code out-discriminates one tuned on execution outcomes. Dockerless beats the strongest open-source verifier — the DeepSWE Verifier — by 14.3 AUC points: 81.0 vs 66.7 AUC on SWE-bench Verified, and 72.1 vs 62.9 on Multi-SWE. Wired into a fully environment-free post-training pipeline as the reward signal, it lifts a Qwen3.5-9B base from 59.6 / 41.3 / 32.3% to 62.0 / 50.0 / 35.2% resolve rates on SWE-bench Verified, Multilingual, and Pro (+2.4, +8.7, +2.9). The SFT story is cleaner still: filtering to the top 25% of a 16K-trajectory pool by Dockerless score beats training on the whole unfiltered pool by +1.8, +6.4, +3.4 points — fewer, better-verified examples winning outright.
Why it matters#
If you've ever tried to stand up an RL or rejection-sampling pipeline for a coding agent, you know the environment is the bottleneck, not the model. Every repo wants its own image, its pinned dependencies, its flaky test harness. That setup cost is exactly why so many "train your own SWE agent" efforts stall before the first reward is ever computed. Dockerless attacks that tax head-on, and its cost accounting is the part I keep rereading: reward evaluation adds only 41–180 seconds per rollout against agent rollouts that already average 2308 seconds — roughly 7% overhead, with no container orchestration at all. If verification is a read over the repo rather than an execution inside it, you can scale your reward signal to any repo you can clone, not just the ones you can Dockerize. That is a genuinely different scaling curve.
The deeper shift is conceptual. The field treats "run the tests" as ground truth, but tests are just one evidence source — and a leaky one, since agents overfit to visible tests (the whole reward-hacking story). Dockerless reframes verification as evidence-gathering: pose the right questions, retrieve grounded answers, judge. If you build sub-agent architectures, this drops in as a first-class verifier sub-agent alongside your mapper and editor agents — and because its verdict cites retrieved spans, it's auditable in a way a green/red test bar never is. It also fits the "gate writes to shared state with grounded verification" pattern cleanly: a read-grounded verifier is precisely the admission control you want on an agent's "done" claim. And it's a concrete answer to the recursive-self-training-collapse work from the same week, which argues you need exogenous verification rather than model-coupled self-review — an evidence-grounded verifier is a plausible exogenous signal.
The caveats#
Compiler signal is the real gap. On compilation-heavy languages the env-free pipeline loses — Rust by 7.0 points, C by 13.3. The authors are refreshingly blunt that "the residual value of env-based supervision is concentrated in compiler signal, not in test execution per se." So the lesson isn't "execution is useless" — it's "test execution is replaceable, compilation isn't yet."
It leans on a reference patch. The verification questions are generated from the issue and a reference patch — which you have at training time but not when the agent faces a novel bug in production. This is a training-pipeline verifier, not a deployment-time oracle.
One base model, one scale. The resolve-rate story rides on a single 9B model; whether the gains hold on frontier long-context models is untested.
Benchmark leakage. AUC on SWE-bench Verified and Multilingual inherits the usual saturation and contamination concerns.
The takeaway#
What I'm filing away: the "run the tests" step in a coding-agent loop is really two things wearing one coat — a test-execution signal and a compiler signal — and only the second is genuinely hard to replace. For anyone building a post-training pipeline, that makes the Docker requirement far more negotiable than it looks; an agentic, evidence-grounded verifier can carry the trajectory-filtering and reward load at single-digit overhead. What I'm doing differently after reading this: when I next design a verification or reward step, I'll split "did it compile" from "did it behave," and reach for read-grounded verification on the behavioral half before I reach for a container.