The model was not the bottleneck
A month of agent sessions that went sideways, a controlled replay of three of them on three models, and the working rules we wrote afterwards.
1The month in failures
Most writing about AI coding agents is about what they can do. This piece is about what went wrong, because that is where the money went.
Over one month, from mid October to mid November 2026, we ran a few hundred sessions with coding agents: Claude Code as the main driver, Codex and other CLIs as background workers for research, review and image generation. The work was a mix you would recognise from any small studio: a phone app, a TV dashboard, a Chrome extension, a font, a recruitment post, a landing page, a thermal printer driver, a rental comparison for a family move, a 3D scene fix, a short explainer video with a cartoon narrator.
We went back through the logs and pulled every session that did not land on the first try. Not the ones with a small follow-up fix, but the ones with long back-and-forth, visible frustration, or a task that was simply dropped. Fifteen sessions qualified. For each one we collected the full log, every file it wrote, the state of the repository, the inputs the human had attached, and the background jobs it spawned, then wrote a short handoff naming what went wrong.
Then we did something we had not done before. We took three of those tasks, rebuilt the starting state, and replayed them from the same first message on three models, several times each, with a blind judge scoring the result against the corrections the human had made in the original session. Twenty-one runs, about $92 including the judging.
The headline result surprised us enough that it became the title of this piece. When the replays started from a clear brief and skipped the heavy machinery, the failures mostly did not come back. The model had been capable all along. What failed was the process around it.
2Six ways a session goes wrong
Each of the fifteen handoffs names one or more root causes. Grouped, six patterns cover all of them. Most sessions show more than one.
Sessions affected, by failure pattern
Done without checking on the real target (6 sessions)
This was the most common failure and the most expensive, because it is invisible until the human tries the thing. The agent writes code, runs whatever check is convenient, and reports success. The check passes. The product does not work where it lives.
A wallpaper and weather widget for a smart TV was built and verified in desktop Chrome. On the TV's own browser engine the layout broke and the weather never loaded. The session ended with the human discovering this on the sofa, remote in hand.
A thermal printer app "worked": it connected, it sent bytes, it reported success. No paper ever came out with the right content, and nobody had asked for a photo of a print. Later it turned out the printer needed Bluetooth Classic, not BLE, which a single real print attempt would have revealed in the first ten minutes.
A camera fix in a 3D scene was judged by reading positions from the DOM. The numbers looked right. The picture was not, and the fix had moved both axes when only one was asked for. Nobody looked at a screenshot until the human did.
The common thread is that "done" was defined by what the agent could check easily, not by what the human would see. A unit test, a DOM measurement, a 200 response: each is a proxy. Proxies drift from reality quietly.
Tool, login or platform blocker found mid-task (6 sessions)
The second failure is mundane, and that is the point. A password manager prompt timed out during a deploy. The agent CLI logged itself out halfway through a long task when a link arrived. Image generation failed through three different routes in a row. A 3D scanner's high-resolution toggle was never mentioned, so an hour of scans came out unusable.
Almost all of these were knowable at minute one. Login state, token scopes, quotas, whether the browser automation is connected, whether the target device is reachable: these are facts about the environment, and they do not change during a task. Discovering them at minute sixty means the agent has built context, made partial changes, and now has to stop, explain, wait, and rebuild. Discovering them at minute one costs a single line.
Guessed instead of searching or asking (5 sessions)
Asked to research short-video ideas for a product, the agent invented a physical kettle to build the ideas around. The real product existed in a project folder on the same machine. One search would have found it.
Asked to write a recruitment post, the agent added a salary figure nobody had provided. The human's rules for hiring copy said, among other things, no salary. Those rules existed only in the human's head and in earlier corrections.
Asked to compare rental places from a pile of chat screenshots and video transcripts, the agent produced a clean comparison with no stated selection criteria. It was well formatted and could not answer the question that mattered, which was "which one should we take?"
Two of these sessions also involved short replies in Vietnamese that could be read two ways. The agent picked one reading silently and built on it. The human had meant the other.
Guessing feels efficient because it avoids interrupting the human. It is the opposite. A wrong guess produces confident work in the wrong direction, and the human has to read all of it before discovering that.
Built more than was asked (3 sessions)
The human asked for a phone app that shows a scrolling LED-style sign and said, in so many words, to make something simple first. The agent ran two rounds of multi-agent ideation, designed a console that would drive several devices at once, and spent two hours on architecture. The session ended with: "it's getting too much complicated." The task was abandoned.
For a short explainer video with a cartoon narrator, the agent generated 26 character faces and 4 test videos before the human had heard a single clean take of the voice, which was the thing that needed deciding first.
Over-building is seductive because it looks like diligence. The tooling makes it easy: ideation skills, brainstorm pipelines, parallel reviewers. Each is useful when asked for. Run by default, they turn a ten-minute task into an afternoon and bury the simple version the human wanted under options they did not.
Handed work back (2 sessions)
In one session the agent asked the human how to test a browser extension, then where, until the human replied: "just do it. don't ask." In another, browser steps were pushed to the human repeatedly until the human pointed out which automated browser the agent already had. The agent had the tools. It did not reach for them.
Non-ASCII text rendered wrong (2 sessions)
A printed label and a TV theme both used fonts that did not cover Vietnamese diacritics. The text looked fine in the code and broke on output. This is a small category with a trivially cheap fix, which is why it belongs on the list: render one sample string in the chosen font and look at it.
Notice what is missing from the list: "the model could not write the code." Not one of the fifteen handoffs named raw capability as the root cause.
3The replay
A list of root causes is a hypothesis. To test it, we replayed three tasks under controlled conditions.
We picked tasks that covered different kinds of work and whose starting state could be rebuilt exactly:
- The signboard app, the abandoned "simple first" phone app. A build task where the original failure was over-building.
- The recruitment copy, scoped to the post text and an image brief. A writing task where the original failure was guessing at unstated rules.
- The rental comparison, from real transcripts and screenshots. A fact-extraction task where the original failure was missing criteria.
A fourth candidate, the 3D camera fix, was dropped because the repository state just before the failure could not be reconstructed. That turned out to be a useful rule in itself: if you cannot rebuild the starting point, the replay measures something else.
Setup
Each run got its own empty sandbox folder with copies of the inputs and the human's original messages, written as a brief with a short shared preamble. Agents ran fully autonomous: permissions bypassed inside the sandbox, but pushing, publishing, the GitHub CLI, the secret store and deploy tokens were all disabled. No run could ask the human a question, because there was no human. After each run we checked that no agent had read the original session's outputs, which would have contaminated the result.
Models: Claude Fable 5 (large), Claude Haiku 4.5 (small), and GPT-6 through Codex (a different provider). Fable and Haiku ran three times per task, GPT once because it drew from a plan quota rather than API spend.
The judge was a separate Fable session that saw each output blind, without the model name, and scored it from 0 to 100 against a rubric built from the human's real corrections in the original session. We hand-checked its two harshest calls. Both held.
4What the numbers say
Judge score for every run
| Model | Signboard | Recruitment | Rental | Cost / run | Time / run |
|---|---|---|---|---|---|
| Fable 5 | 100 | 100 | 90 (83 to 94) | $3.46 | 5.1 min |
| Haiku 4.5 | 81 (73 to 87) | 83 (68 to 91) | 59 (53 to 67) | $0.22 | 2.6 min |
| GPT-6 (Codex) | 100 | 94 | 97 | plan quota | 5.9 min |
Mean of 3 runs for Fable and Haiku, range in brackets. One run for GPT.
The original failures did not come back
This is the finding that matters most. In the original signboard session, the agent spent two hours on ideation and a multi-device design before the human gave up. In the replay, all seven runs, across all three models, built a simple single-device app. All three Fable runs explicitly noted that they skipped the ideation step because the human had asked for simple first.
In the recruitment task, every run kept salary out and wrote in the right language. The rules that had been broken in the original session were in the brief this time, so they were followed.
Put plainly: the same model that failed in the original session succeeded when it started from a clear brief without the heavy pipeline. Clarity of the request and restraint in the process did more for quality than the choice of model did.
Large against small
On the app and the copy, the small model was behind but usable. On the rental comparison, the gap was wide: 90 against 59. That task required pulling facts out of messy transcripts and chat screenshots, and that is where the small model started inventing. It placed a street in the wrong district with nothing in the inputs to support it. It ticked amenities nobody had mentioned. It missed one of the six places entirely in two of three runs. In one run it wrote mostly in English despite the human saying they preferred Vietnamese content. All three of its recruitment drafts used punctuation the human's style rules ban.
The large model cost about 16 times more per run. Whether that is worth it depends on the job, which is the point of routing (section 5).
One provider against another
GPT matched Fable on quality and produced the two most useful findings in the whole exercise. First, it noticed that a search engine's cached snippet of a related job description still showed the client's name, even though the live page had been cleaned. Every Claude run that reported on that check said the copy was clear to post. Second, it noticed that one of the job descriptions required Vietnamese, which conflicted with the English-only instruction for the post. Nobody else flagged it.
Its weak spot was environmental: it could not launch a browser inside the Codex sandbox, which it reported plainly instead of pretending.
The lesson is not that one provider is better. It is that a second provider looks at the same material differently, and on research and review that difference is worth paying for.
Same model, three runs
Fable was steady. Its one dip was a rental run that scored 83 because it stated no selection criteria and left its open questions unprioritised: the same failure as the original session, appearing once in three tries. Haiku swung by 14 to 23 points within a task. Roughly one run in three was noticeably worse than the others.
That has a practical consequence. With a small model, a single good run is not evidence that the next one will be good. If you route work to it, you need a check on every output, not a spot check.
5The process
Here is what we changed. Each rule traces to a failure pattern above and, where possible, to replay evidence. The agent-facing version, written tersely for agents to load every session, is in agent-workflow-rules.md.
1. Done means proven on the target.
An agent may not report done from tests, DOM numbers or "should work." It reports done with evidence from where the output lives: a screenshot at real phone width for a mobile page, the device or its real browser engine for TV work, a photo or the printer's own preview for a print job, real data for a dashboard, before-and-after screenshots for a visual fix. If the target is unreachable, the report says "built, not verified on X because Y," and never rounds that up.
This one rule would have caught six of fifteen sessions. It costs a few minutes per task. We made the existing Proof line in our report format require target evidence, so the rule is enforced by the shape of the final message, not by memory.
2. Preflight before long tasks.
Before any task expected to run past fifteen minutes, check the environment the task will touch: agent login, token scopes, the local secret copy, secondary model quota, browser automation, target device reachability, and font coverage if the output renders non-ASCII text. Report gaps in one line, then start. Six sessions hit a blocker that existed before the first message.
3. Search first, then ask once.
Before building around an unfamiliar name, search the workspace and notes, then the web. Never fabricate a product, address, amenity or spec; unknown stays marked unknown. For work over an hour or with a human audience, ask one structured question up front, with tappable options and a recommendation, covering only what changes the output: audience and language, what "good" means, and the target surface. Short replies that can be read two ways get restated in one line before acting on them.
The trade we are making is one interruption at the start in exchange for zero wrong-direction hours later. The replays support it: the recruitment rules, once stated, were followed by every model, every time.
4. Simple first, by default.
The first deliverable is the smallest thing that works end to end on the real target. No ideation rounds, brainstorm pipelines, multi-agent reviews or "10x" expansions unless the human names them. Change only what was asked; a fix to one axis does not touch the other. For subjective choices like a voice or a look, produce two or three candidates and get a pick before producing twenty.
The replays are the strongest evidence in this piece: without the pipeline, every model got the signboard app right.
5. Do the work yourself.
If the agent has a browser, a CLI or a device API, it uses it. "How do I test this?" is the agent's question to answer. When a step truly needs a human (a physical action, a login with a second factor, a payment), the agent says exactly what, why, and what it will do next, in one message rather than a drip of them.
6. Preferences become checks.
Every correction from the human becomes a one-line rule in a feedback log next to the skill it belongs to, read before the next round of that kind of work. A rule stated twice becomes an automated check: a style linter on every deliverable, a diacritic render test for any font choice, a checklist the hiring-copy skill loads before drafting. Checks run before reporting, every time. The small model broke the punctuation rule in all three of its recruitment drafts; a linter would have caught all three for free.
7. Route models by the job.
| Job | Route | Why |
|---|---|---|
| Facts from messy inputs, research, anything the human reads | Large model | Steady across runs; the small model invented facts here |
| Second opinion on research or review | Other provider | Found issues every same-provider run missed |
| Small, fully specified, checkable jobs | Small model, then verify | About 16 times cheaper, about 1 run in 3 noticeably worse |
| Browser-driven testing | An agent with a real browser | Sandboxed CLIs may not launch one |
8. Test rule changes before trusting them.
A rule that sounds right can still make things worse. Before adopting a change to the rules, a skill or a system prompt, replay two or three past failures with it and compare. The next section describes how.
6Running your own replay
The replay setup cost about $30 per task for three models with judging, and each task finished in under fifteen minutes of wall time. That is cheap enough to run every time you change how your agents work. Here is the recipe, including the mistakes we made.
- Collect the failures. Go through a month of session logs and pick sessions with long back-and-forth, visible frustration or abandonment. For each, keep the full log, the files it wrote, the inputs, any background jobs it spawned, and a repository snapshot (a git bundle of all refs plus the uncommitted diff and untracked files).
- Write a handoff per session. What was asked, what went wrong, the human's exact corrections, and what "done" would have looked like. The corrections become the rubric.
- Pick replayable tasks. Only tasks whose starting state you can rebuild exactly. Scope each down to what can be judged without hardware or a human in the loop.
- Sandbox each run. One folder per run, inputs copied in, no access to the original outputs. Disable anything that leaves the machine: push, publish, email, deploy tokens, the secret store. Disable the agent's ability to ask questions, since nobody will answer.
- Run each model three times. One run per model hides variance, and variance was one of our most useful findings.
- Check for contamination. Grep each run's transcript for reads of the original session's files before trusting its score.
- Judge blind, then check the judge. Strip model names, score against the rubric, then read the harshest and most generous calls yourself.
- Show outputs side by side. A simple HTML viewer with one column per run, the score and the judge's reasons on top, made the patterns obvious in minutes.
Mistakes worth avoiding
Judging by keyword. An automated check marked one run compliant on "mentions expat candidates" because the word appeared in the post. It appeared in an unrelated section about a different office. Judge by reading.
A rubric that is too easy. The signboard rubric could not tell the top models apart; three of them scored 100. A rubric needs at least one criterion that a good run can fail.
A same-family judge. Our judge was Fable, which may favour Claude-style output. GPT still scored level with Fable, which is reassuring, but a judge from a different provider, or two judges, would be stronger.
Running out of budget mid-judging. About $7 of our judging was wasted on runs that hit a usage limit and returned nothing. Cap per-call spend and make the judge script resumable.
7Limits
Three tasks out of ten replayable ones is a small sample. Three runs per Claude model and one for GPT means the GPT numbers are promising, not settled. The judge shares a family with two of the three models. The failure patterns come from one person's month of work, with that person's habits and that person's kinds of task. Your distribution will be different.
What we are more confident about is the direction. When the brief was clear and the process was simple, the failures that cost us the most did not recur on any model. That matches what the handoffs said independently, and it matches the cost: the expensive sessions were expensive because of hours spent in the wrong direction, not because a model was slow or a token was dear.
8The rules file
The agent-facing version of everything in section 5 lives in agent-workflow-rules.md. It is short on purpose, because agents load it at the start of every session and every line costs context. It contains the preflight checklist, the "ask once" scope, the simple-first defaults, a table of required proof per output type, the feedback-log format, the routing table, the report shape and the replay recipe. Drop it into a global instruction file, or reference it from one.
The cheapest token is the one spent on the right task. Most of ours were spent on the wrong one, confidently.
Get the full research
This page is the short version. The full write-up has the per-session breakdowns, all 21 replay outputs side by side with the judge's reasoning, the rubrics, and the scripts we used to run the replay. Leave your email and I'll send it to you.