Most mornings the arxiv list hands me another benchmark. Today it handed me a number I could not stop turning over: an agent following its own skill file performs only 56% of the steps that file mandates — and still produces artifacts that pass the output checks. I have written a lot of skill files on the quiet assumption that writing a step down is roughly the same as the step happening. This paper says that assumption is off by about half, and it proposes a fix. The thesis: prose is an authoring surface, not an enforcement surface.
What it does#
SIGIL: Compiling Agent Skills into Typed Harnesses, from Jayanaka Dantanarayana, Savini Kashmira, Lingjia Tang and Jason Mars, goes after what they call skill compilation. A skill today is a prose procedure file loaded into the model's context and executed by a tool-calling loop. It is described to the runtime but never encoded in it, so the model re-derives the control flow on every single run — and any run where it re-derives it slightly differently silently drops a step. Mandated verification is exactly the kind of step that gets dropped, because skipping it costs nothing visible.
The known remedy is to write a harness: put the procedure into program structure so the loop physically cannot skip it. Almost nobody does, because hand-writing harnesses is tedious and it throws away the prose authoring surface that made skills catch on in the first place. SIGIL compiles one into the other. At its center is AG-IR, a typed agentic intermediate representation whose organizing question is an Owner test: is this step's output a function of its inputs? If yes, the step is code-owned and executes unconditionally. If no, it needs judgment and stays model-owned. Nodes come in four families — Mind nodes graded from tightest to loosest (Gen-Enum for fixed-set classification, Gen-Fill for typed objects, Gen-Edit for mutations, Gen-Raw for free text), Flow nodes (Route, Loop, Spawn, Call), Boundary nodes that split idempotent reads (Sense) from writes and deliverables (Act), and pure Code nodes. Modality — mandatory, forbidden, discretionary — stops being an adjective inside a sentence and becomes program structure. Every node keeps provenance back to the skill sentence it came from, so the compiled artifact stays auditable against its source.
The key result#
Thirty skills across three families (document and tooling, software process, governance and compliance), three process scenarios per skill, three repetitions, on gpt-4o and gpt-5. The metric is Applicable-Mandate Compliance: of the mandated steps a run actually reached, how many did it perform, judged against a typed reference procedure. Prose scores 56% on gpt-4o; the compiled harness scores 86%. Full-procedure completion goes from 28% to 65% — 2.3× as often. And it is cheaper, not more expensive: median token ratio of 0.58× against prose. Across all thirty skills the harness matches or beats prose, with 28 strict wins, 2 ties and zero losses. But the number that actually matters is the one across model generations. Moving gpt-4o → gpt-5, prose climbs from 56% to 68%. The harness does not move: 86% to 86%. Structure substitutes for capability, and the substitution is flat.
Why it matters#
The finding that should bother anyone shipping skills is the pairing: 56% of mandated steps performed, artifacts still passing the output checks. That is the reward-hacking shape again, but relocated from the code to the procedure. An output check tells you the deliverable looks right; it tells you nothing about whether the run took the path you specified. If your skill says "run the test suite before reporting done" and the report reads fine, you have no signal at all that the step was skipped — and the failure mode is invisible precisely on the runs where it matters. Which means most of us are shipping procedure documents whose compliance rate we have never once measured. The cheap version of this experiment is available to anyone today: pick the mandated steps in your own skills that leave a trace (a command executed, a file touched, a tool called), and count how often they actually appear in the transcript. My guess is the answer is nearer 56% than 100%.
The Owner test is the useful export even if you never run SIGIL. Walk a skill line by line and ask, for each step, whether its output is a function of its inputs. Everything that answers yes should not be prose at all — it should be a script the agent invokes, a hook, a pre-commit gate, a subagent with a fixed contract. Prose should retain only the genuine judgment. That reframing also changes the upgrade calculus: teams routinely treat prose-reliability problems as something the next model will fix, and this paper prices that bet at 56 → 68 over a full model generation, against 86 available today from structure. The token result kills the last objection too — harnesses are usually assumed to be heavier, and they come in at 0.58×, because re-deriving control flow on every run is itself expensive. Cheaper, more reliable, and stable across model swaps is an unusual combination to find in one column.
The caveats#
Two OpenAI models only. The shape of the prose curve across a wider capability range is genuinely open — a frontier model with stronger instruction adherence could narrow the gap, and the authors say so.
The 30-skill sample leans document, process and compliance, which are mechanism-heavy by construction. A skill that is mostly judgment has proportionally less to compile, and the paper's own breakdown shows judgment-heavy skills retain a large model-owned residue.
Scoring convention matters here: the judge auto-credits code-owned gates from the node trace. The authors recompute using only manually-judged mandates and get 77% vs 56% — still a large gap, but 21 points rather than 30.
AMC excludes mandates the run never reached, which sensibly avoids branching-adjudication bias but also means "the agent bailed early" does not penalize the headline number the way skipping a reached step does.
Compilation is itself an LLM step, so fidelity of the harness to the prose intent is a new error surface. Provenance makes it auditable; it does not make it correct.
The takeaway#
What I am filing away: a skill file is a proposal, and the gap between proposal and execution is roughly half. It is also a good week for that lesson — a separate ablation posted the same day (2607.27250) found that persistent context files like AGENTS.md and CLAUDE.md do not measurably move correctness across 288 runs on two frontier agents, because the failures are implementation skill rather than missing repository knowledge. Two papers, one direction: prose you hand an agent is weaker than it feels while you are writing it. What I am doing differently is running the Owner test over my own skills and pulling every deterministic step out into something the agent invokes rather than something it reads. Verification steps go first — they are the ones the model is most inclined to skip, and the artifact will never rat them out.