Friday's cs.SE batch had the usual pile of benchmarks, and then one title that read like an accusation: Natural-Language Workflows Are Not Software Yet. I have a directory of markdown files that agents execute as procedures — skills, CLAUDE.md, sub-agent briefs — and I have been treating them as code without ever admitting it. This paper puts numbers on what that costs. The thesis I came away with: the unreliability of prose workflows is not a prompt-quality problem, it is an undeclared-state problem.
What it does#
The paper introduces Artic, an artifact-driven workflow compiler (arxiv 2608.21341, Xu, Guo, Shen, Cheng and Zhang, submitted August 21). It takes a workflow written in ordinary English and compiles it into a representation where each step explicitly declares the artifacts it reads and writes, where semantic constraints gate the artifacts a step produces, and where branching happens through explicit control transfers rather than sentences the executor has to interpret. The framing is the interesting part: instead of asking whether the prose is clear, the compiler measures the enforcement burden that the workflow dumps on the agent at runtime.
It measures that burden three ways — cyclomatic complexity over the finest-grained control flow, context pressure (accumulated token cost along a path, taking the max across branches), and compaction pressure, which uses live-variable analysis to count how many artifacts must survive a context-compaction boundary. Steps that score badly get decomposed through a constrained optimization loop: the LLM proposes a rewrite, a constraint checker hands back diagnostics, repeat until the step fits. Because the transformation is itself LLM-assisted, Artic does not trust it. Faithfulness checking is decomposed into local obligations, and then scenario-based dry runs enumerate coverage-guided test cases and have validators simulate the original and compiled workflow side by side, comparing trajectories. That is the split from prior work: this is neither "rewrite the prompt better" nor "just generate code," it is a compiler with an optimizer and a test suite.
The key result#
Across 488 problem instances drawn from 11 real-world domain workflows (SOP-Bench and the χ-Bench healthcare simulator — customer service, warehouse inspection, dangerous goods, patient intake, know-your-business, averaging 9.4 nodes and 9.2 artifacts each), compiled workflows resolved 85% of tasks versus 62% for the original text — a 28 percentage point gain, holding at 82–85% across executor models from GPT-OSS-120B to Qwen3-235B. Skill rewriting got 66% and code generation 73%; MASFactory scored 0% and Chat2Workflow 32%. But the reliability numbers are the ones that stuck with me: under repeated execution, pass^10 was 72% for compiled workflows against 16% for text, cross-model consistency went from 48% to 80%, and average per-agent input tokens dropped 63%. Under perturbed inputs the compiled workflows held 100% of their performance while text degraded to 80%.
Why it matters#
The diagnosis is what I actually want from a paper like this, and it names two failure modes precisely. The first is implicit data dependency: when a step says "using the results from the previous step," the executor has to infer which prior result is meant, and every one of those inferences is a coin flip that compounds down the chain. The second is context pressure on long or branching instructions. I have hit both, repeatedly, and I had been filing them under "the model got confused." Compaction pressure is the metric I had not thought to name — the number of artifacts that must survive a compaction boundary is a static property of the workflow that you can compute before you run anything. That reframes a whole class of flaky long-running agent runs from a runtime mystery into something closer to a lint check.
What I would actually do with this: not install Artic — it is a research artifact evaluated on business SOPs — but steal the representation by hand. Take your longest skill, walk each step, and write down what it reads and what it writes. The steps where you cannot name the artifact are the steps that break, and that exercise costs an afternoon. The code-generation baseline at 73% is the other useful signal: turning a workflow into a script recovers most of the gap without any of Artic's machinery, which suggests a good chunk of the win is simply making state explicit, whatever syntax you make it explicit in. And the ablation deserves its own note — dropping faithfulness validation cost 16 points, meaning the LLM-generated workflow needed tests as much as the original did. Anyone auto-generating skills with a model should read that as a warning, not a footnote.
The caveats#
The domains are business SOPs and healthcare intake, not code editing. There is no SWE-bench-style task anywhere in this evaluation. Everything I said above about skills and sub-agent briefs is my extrapolation, not their measurement.
The 28-point headline is against raw text. Against the strongest baseline they report — code generation — the gap is 12 points (85 vs 73), which is a much more ordinary result.
pass^k at k=10 flatters any variance-reduction result almost by construction. The 16% figure for text mostly restates that prose workflows are nondeterministic, which was not in dispute.
Compile cost is absent from the headline numbers. There is an iterative LLM optimization loop plus a multi-pass validation stage per workflow, and the paper does not price it against the 63% runtime token savings.
Judgment-heavy steps get no benefit, and the authors say so plainly — in the know-your-business domain, deciding whether an address mismatch is a typo or fraud depends on the executor model's calibration, not on workflow structure.
The takeaway#
I am filing away the three burden metrics as a review checklist for my own skills, and more than that, the reframe underneath them. Writing a workflow in prose is not a lighter-weight alternative to writing code — it is writing code with all the declarations deleted, and then paying for those deletions at runtime, nondeterministically, in a place where the stack trace does not point back to the omission. Concretely: before I add another step to a long skill, I am going to name the artifact it reads and the artifact it writes. If I cannot name either one, I have found the bug before running it.