AI · 135 POSTS · PAGE 3/14
AI.
Writing tagged AI: 135 posts on AI systems, engineering tradeoffs, and building products that have to work in production.
Across 147,002 coding-agent traces, early-failure predictors were at chance within a task (AUC 0.50), and plain continuation beat restart on solve rate and cost. Don't abort on prefix signals.
Across 3,000 turns, five models in Codex and Claude Code churned up to 31x the original code, and often kept editing after declaring the task done. Cap loops by diff budget, not the agent's verdict.
Deleting old turns rarely hurt SWE-bench results, but agents re-fetched what was removed, and under cache-aware pricing none of 42 setups got cheaper.
Agents reached sufficient evidence of a blocker in 98.6% of runs, then kept working. A simple escalation tool and a harness monitor cut wasted steps and unsafe requests.
In an 18-condition requirements pipeline study, specialised prompts drove the F1 gain (up to +0.075) while the typed handoff moved it by -0.003 to +0.013. Schemas pay off as fault detection, not accuracy.
PAUSE intercepts a coding agent's first source edit and injects one plan, lifting SWE-bench Verified by up to 16.8 points at about a tenth of a periodic overseer's Planner cost.
Coding agents followed an outdated instruction file in 90.1% of choices even when they flagged the conflict; one user-level sentence flipped it to 91.4% following the code.
RegretBench shows coding agents rarely back up before destructive steps, and capability gains don't fix it. A cheap interception layer, BackUpper, restored 92.2% of state at $0.014 per task.
A self-improving agent platform's LLM gate silently ignored its input, and regenerated rollbacks failed to restore behaviour. Two cheap checks catch both before you let changes ship unattended.
SWE-Residual shows five frontier agents lose 20.6–53.5 points when they inherit a predecessor's failed attempt, mostly by trusting it. If your agents resume work, test for that.