Skimming this morning's arxiv drop, most of the coding-agent papers were benchmarks — new suites, new leaderboards, one more resolve-rate number to squint at. Then a dataset paper stopped me, because it isn't measuring whether a model can close a task in a sandbox. It's measuring what real developers physically do to AI-generated code after they accept it: keystroke by keystroke, in their own editors. My thesis for this post: the manual edit — not the Git commit, not the SWE-bench pass — is the most honest training and evaluation signal we have, and we've been ignoring it because it was hard to collect.
What it does#
DECODE (Developer Edits of Code Dataset), from Jenny T. Liang, Mihika Bairathi, Wayne Chi, Ameet Talwalkar, Nishant Subramani, and Valerie Chen, is a corpus of 53.6K real-world in-IDE edits of AI-generated code in Python, TypeScript, and JavaScript, sourced from over 1,000 developers. The methodological move is the whole point. Git commits — what nearly every code LLM trains on — only preserve the final, successful snapshot: the polished diff that got merged. They discard the messy middle, the part where you accepted a completion and then immediately started hacking it apart. DECODE instruments the editor to capture exactly that middle: the accepted suggestion, and every edit that followed it, as a trajectory.
That reframing gives the paper two lives. First, it's an observational study — a look at when, why, and how AI-generated code actually gets rewritten in the minutes after acceptance. Second, it's a benchmark: given an accepted completion and its surrounding context, predict the edit a developer will make next. Prior 'code editing' data was mostly synthetic or reconstructed from commit history; DECODE is the raw, granular record of humans repairing model output in situ.
The key result#
The headline finding is that acceptance is not the same as approval. Most editing happens within the first 15 minutes of accepting a completion, and — the number that stuck with me — 31% of edit trajectories end in the AI completion being removed entirely. Nearly a third of accepted suggestions are edited into oblivion, fast. The second result is the benchmark kicker: finetuning an open-source 3B model on DECODE beats frontier LLMs at predicting those edits — a tiny specialist trained on the messy middle out-predicts the giant generalists trained on clean endpoints.
Why it matters#
The accept click is a label we've quietly been trusting, and this dataset says the label is noisy in a specific, measurable way. If you build an in-editor assistant and your north-star metric is completion acceptance rate, you're optimizing a number that reverses itself in nearly a third of cases before the coffee is cold. The real signal isn't the accept — it's what happens in the 15 minutes after. This is the keystroke-level cousin of the 'developer pushback' framing from the large-scale misalignment work: overrides and corrections, not pass rates, are where the truth about an assistant's usefulness lives. DECODE just pushes that resolution down from the PR to the individual completion, and gives you a concrete rate to instrument against: does this suggestion survive the next quarter-hour?
The finetuning result reframes the data problem, too. A 3B model beating frontier models here isn't a scaling story — it's a distribution story. Frontier models learn from commits, which are endpoints; edit prediction is about trajectories, the intermediate states those models never see. For anyone building an in-editor agent, the practical move follows directly: instrument your own edit trajectories and treat 'what will the developer change about this?' as a first-class prediction target, because a cheap specialized model trained on that distribution can beat reaching for the biggest general model by reflex. It's the empirical case for developer-centric data collection over scraping yet another terabyte of GitHub — the value isn't in more clean code, it's in the repair signal you're currently throwing away.
The caveats#
Opt-in population. 53.6K edits from 1,000+ developers is large, but it skews toward whoever enabled the telemetry; the exact tooling and developer population aren't fully characterized in the abstract.
Three completion-friendly languages. Python, TypeScript, and JavaScript are autocomplete-heavy. Systems languages, and long-horizon agent edits over many files, may show very different patterns than in-line completion repair.
Removal isn't blame. A completion deleted within 15 minutes might be a bad suggestion — or a developer changing their mind. The 31% measures deletion, not fault.
Edit prediction is a proxy. A 3B model winning at predicting edits doesn't mean it writes better code; it means it models this specific distribution better — the same coverage-not-precision caution that shows up elsewhere in agent evals.
The takeaway#
What I'm filing away: stop treating acceptance as the label and start treating the edit as the label. If I'm evaluating an in-editor assistant, I want a 'survives-15-minutes' metric far more than an accept rate. And if I'm collecting data to improve one, the messy middle of real edit trajectories is worth more than another mountain of clean commits. What I'm doing differently after reading this — adding edit-trajectory capture to the eval harness, and treating a small, finetuned edit-predictor as a plausible component rather than defaulting to the largest model in the drawer.