I built a small software factory. Here is how it works and where it failed

TL;DR. I built a small software factory on top of IndyDevDan's open-source Super Simple Software Factory: a planning agent, a building agent and a set of deterministic gates that take one written slice of work from a spec to a commit. The design is his. The gates in section 8 are what I added. In six days I ran it 47 times for $95.82. Of the 32 runs in its strictest lane, 21 passed on the first loop, 7 passed after one fix loop and 4 failed. Eleven of those runs met a red gate, and the gate was right six times and wrong five. The part I most need to remember is the other direction: 15 runs that cleaned up our tests passed every gate, and 13 of those had still left tests asserting less than before. Breaking the code under those tests is what found them. Every gate the factory has today exists because a named run failed without it. So the factory is one half of a loop: it builds, something outside it doubts the result, and what the doubt finds becomes the next gate.

I am writing this down for myself, in public. I am not selling a factory, and I do not think you need one. I think the measurements are worth having, including the ones that do not flatter the thing I built.

Some context first. Hearso is a small real-time multiplayer trivia platform. I ship it in batches, and the batch this post comes from changed nothing in the product: it turned on stricter TypeScript, added lint rules for our React tests and cleared everything they reported. That is hundreds of files of careful, repetitive work, which is exactly the work I wanted agents to do under supervision.

What this post covers

1. What I mean by a software factory
2. How one slice moves through it
3. The agents, and what each may touch
4. The first two days, when it failed five different ways
5. The numbers from 47 runs
6. What the gates caught, and when the gates were wrong
7. What passed every gate and was still wrong
8. Every gate has a run it came from
9. Where the terminal fits
10. What is not built, and what I would not claim
11. Closing the loop

1. What I mean by a software factory

The factory is called OGSF, for One Generalist Software Factory. It is a folder of Python in the repository. It began as Super Simple Software Factory, an open-source project by IndyDevDan under the MIT licence, which I installed on 2026-10-08 and have been changing since. The design is his: deterministic Python owns the sequence, and coding agents are bounded steps inside it. What I added in these six days is the table in section 8, each row built after a run failed without it. Git ignores that folder, so the factory's code lives only on my machine and travels with it. That turned out to matter, and I come back to it in section 6.

I set it up for two reasons. I need to track every agent that runs in my local development workflow, and I need each agent kept to the one task it was given. A terminal session gives me neither by default. One agent plans, edits, tests and commits in the same conversation, and what is left afterwards is a scroll of text.

Running it for six days gave me more reasons than I started with. Each one comes back later in this post with its numbers:

• A record that adds up. Every run writes its cost, its time and its phases to a database. That is the only reason this post can say $95.82 (section 5). My terminal sessions over the same days recorded no cost at all (section 9).
• Limits that are enforced, not requested. Each agent has a list of what it may touch, and code compares the tree before and after it ran (section 3).
• The same checks in the same order. No gate is skipped because I am tired or in a hurry (section 2).
• It refuses before it spends. Preflight turns a run away before the run has an id, so a wrong branch or an unclean tree costs nothing (section 2).
• Repetition without starting a new conversation each time. Seventeen runs did the same kind of test cleanup from specs of the same shape (section 7).
• A failure turns into a check. Every gate was added after a named run failed without it, and the factory's own tests grew from 9 to 85 (section 8).

None of these says the factory was faster or cheaper than working by hand. I did not measure that, and section 10 says so.

The name is old. I did not invent the term, so I went and read where it comes from. A 1997 survey of the idea, The Software Factory: Contributions and Illusions, traces it to 1968, when R. W. Bemer suggested that General Electric build a software factory with "standardized tools, a computer-based interface, and a database with historical data for financial and management controls". Around the same time M. D. McIlroy of AT&T proposed a different picture under a similar name: build new programs out of reusable parts. By the survey's account, Hitachi was the first company to call a software organisation a factory, in 1969. System Development Corporation in the United States followed in 1975 and 1976, and NEC, Toshiba and Fujitsu in 1976 and 1977. In 2004 two architects at Microsoft, Jack Greenfield and Keith Short, used the name for something else again: a kit of languages, patterns, frameworks and tools for producing variants of one kind of application (their book).

Two things in that history are useful to me. The first is that Bemer's list from 1968 already holds my first reason. A database with historical data is what lets me add up 47 runs. The second is the survey's warning about the word itself. It says "factory" carries "the controversial connotation that software development and maintenance is comparable to mass-production of industrial products, and arguably this is not the case", and that this "can easily lead to illusions". I read that as a caution to me, and section 7 is where it applies.

What agents change. In those accounts a factory is an organisation of people with shared tools and procedures. The newest use of the name puts coding agents where the programmers were. The most public example I know of is StrongDM's, which Simon Willison wrote up in February 2026. Their team has two rules: "Code must not be written by humans" and "Code must not be reviewed by humans". People write specs and scenarios. The scenarios are often stored outside the codebase, the way a holdout set is kept apart in model training. Their own yardstick for spending is "$1,000 on tokens today per human engineer". Willison ties this to what Dan Shapiro calls the Dark Factory, where no human looks at the code, and he asks in the same piece how reviewing nothing could be a sensible strategy.

Mine sits at the other end of that range, and I want to be plain about it. The factory writes code. It does not get to say the code is good, and reading what it wrote is still my job. All 47 runs together cost $95.82, less than a tenth of that yardstick for a single day. Section 7 is why I keep the reading.

It does one thing. I write a spec for a slice of work, I start a run, and the factory tries to turn that spec into a single commit on a feature branch. It never pushes, never opens a pull request and never merges. Those stay with me.

Two sentences in its code say what it is trying to be. Both are IndyDevDan's, from the project it began as. The first is in the module that runs lint, the type check, the tests and the build:

A known command is not a judgement call. Anything whose invocation you can write down belongs here as code — it runs in milliseconds, costs nothing, and returns the same answer every time. Agents are for the parts that need reading and deciding.

The second is shorter: "Agent proposes, code disposes."

So an agent never decides whether its own work passed. An agent writes a plan or changes files. Plain code, with no model in it, decides what happens next. Each run is a sequence of phases, and a phase is one of three kinds: the engineer (me, stating the request), an agent, or code. Every phase starts out marked as failed and has to earn success.

It is also worth saying what it is not. It is not a sandbox: the agents run in my working tree with my environment. The only containment is a check after the fact that compares the tree before and after an agent ran and rolls back writes outside what that agent is allowed. And it runs one slice at a time, because two runs would be writing to the same working tree.

2. How one slice moves through it

There are several lanes, from a one-prompt question to a full plan, build, review and document chain. Almost everything in this post went through the strictest one. Its own summary of itself is three lines:

Phases: engineer(request) -> planner -> [stop here on a noop/blocked verdict]
        -> code(baseline) -> builder
        -> [code(contract) -> code(verify) -> builder(fix)] bounded -> git(commit)

Figure 1. One slice through the strict lane. Purple boxes are agents, blue boxes are plain code. The fix loop runs at most twice, so the contract and the gate are tried at most three times.

Start from the end. For me to accept a slice's commit, it must change only what the spec said it could, meet the spec's criteria and break nothing that worked before. Those are the scope check, the spec's own checks and the quality gate. Every other step makes them cheaper to reach or cheaper to fail.

Here is each step, in the order it happens.

Preflight. Before a run even has an id, a script checks eight things: the working tree is clean, I am on a feature branch, the spec's file name matches the branch name, the spec is committed, no pull request from this branch has already merged, the spec's contract can be read, the contract names this lane, and the contract's own probe says there is still work to do. If any check fails it prints the list and stops with "Nothing ran, no tokens spent."

Request. My prompt is recorded. For a slice, the prompt is the spec file.

Plan. The planner agent reads the spec and the code and writes a plan. It also returns a verdict: implement, no-op, or blocked. Anything but "implement" ends the run right there, as not accepted. A dispatched slice that produced nothing is something I have to see, so it is not a quiet success.

Baseline. Before the builder touches anything, code runs lint and the type check and stores every diagnostic it finds.

Build. The builder agent implements the plan. It hands back an envelope: a status, a summary, the files it says it changed, and the whole commit message.

Contract. Code runs three things in order. First the repository's own pre-commit hook, as a rehearsal, because the commit is the last step and a hook that refuses the tree would otherwise be found only after every paid phase. Then scope: every changed path must match a pattern the spec declared. Then the spec's own checks, which are shell commands that must exit 0.

Verify. The quality gate: lint, the type check, the unit tests, and the build if the type check passed. Lint and the type check are compared with the baseline, and I explain why in section 4.

Fix. If the contract or the gate is red, the builder gets the failing command's output, word for word, and tries again. This can happen twice. A third red result ends the run with nothing committed and the tree left as the builder left it.

Commit. One commit of the whole slice, made by the lane and not by an agent.

The spec carries its contract in a fenced block. This is the top of a real one, for a slice that cleared type errors in one folder:

lane: sdlc-strict
precondition: |
  files=$(git ls-files --cached --others --exclude-standard lib/pond | grep -E '\.(ts|tsx)$')
  out=$(printf '%s\n' "$files" | xargs node scripts/type-safety/strict-flag-errors.mjs || true)
  line=$(printf '%s\n' "$out" | head -1)
  printf '%s\n' "$line" | grep -q -E '^strict-flag-errors: [1-9][0-9]* errors in '
writes:
  - lib/pond/*
  - changelog/unreleased/strict-flags-lib-pond.md
checks:
  - name: lib/pond has no error with both flags on
    run: |
      ...

The precondition is "there is at least one error left to fix here". The writes list is the only ground the run may change. Each check is an acceptance criterion written as a command. The module that reads this block says why it exists: "A slice spec is prose for agents. Prose cannot refuse a dispatch, and by the time an agent has read it the tokens are spent."

Figure 1 draws one loop, from a red result through a fix and back. A second is not in the picture: a run that fails in a new way comes back as a new gate (section 8). The two boxes that are not code are the agents.

3. The agents, and what each may touch

The strict lane uses two agents. The factory has five configured.

Agent	Its one-line purpose	May write
planner	"Turn a request into a plan the builder can implement without asking questions."	the specs/ folder only
builder	"Implement the plan exactly; report every changed file in the envelope."	anything except the factory's own code
scout	"Find and report where things live; change nothing."	nothing
reviewer	"Confirm that what was built is what was asked for; change nothing."	nothing
documenter	"Write up the change that was just made, from the diff; document only."	docs and Markdown

There is no tester agent, on purpose. Running the suite is a known command, so it is a code phase.

Both strict-lane agents run through one coding-agent program, with the model, the thinking level, the tools and the system prompt set per agent in one configuration file. The planner ran on anthropic/claude-sonnet-5-5 throughout. The builder started on openai/gpt-5.3-codex and moved to the same Anthropic model on 2026-10-11, when the OpenAI account ran out of credits in the middle of a run. I say more about that switch in section 5, mostly to say what I cannot conclude from it.

Three files are protected from every agent: the factory's modules, its configuration and its lane scripts. The quickest way to pass a check is to change the check, so the configuration says: "An agent must not be able to edit the machinery that decides whether its own work passed."

That rule is not theoretical either. A comment in the permissions module records that an agent once ran git checkout on the factory's folder, "discarding uncommitted changes to the very quality check it was about to be judged by". A list of allowed tools is a capability list, not a sandbox, because one of the tools is a shell.

Every agent's system prompt also ends with a section of project conventions filled in from the configuration. The module that renders it explains itself: "a convention an agent is not told about is one it will 'fix'". Section 8 has the incident behind that sentence.

4. The first two days, when it failed five different ways

I would like to say I designed the lane in section 2. I did not. On the first day the factory had no preflight, no contract, no baseline and no verdict. Each of those was added after a run went wrong, and the first two days supplied five such runs.

It did not start. The first command I ran after installing it failed with ImportError: cannot import name 'GSFConfig'. One module in the template imported a name that was spelled OGSFConfig everywhere else, and the import ran at load, so every entry point failed the same way. The fix was two lines, here and in the template. The first two runs that worked were a one-line summary of the repository (54,050 tokens, $0.037) and a read-only tour of its folders (64,286 tokens, $0.036).

The rollback fired on bytecode. The first attempt at a real slice was aborted by the permissions check, which saw fourteen deleted .pyc files in a cache folder under the protected path and called it a breach. Nothing an agent did was wrong. I narrowed the protected pattern to the Python source files.

It could not pass on a codebase that was already red. This is the expensive one. Session 4b06828b was a slice that moved our environment variables behind a typed module. At the time, the project already had 1,735 type errors from the stricter flags. The gate was all or nothing, so it compared the tree with zero. The run built, verified, fixed, verified, fixed and verified again. The type-check count fell from 1,735 to 1,292 along the way, which is real progress, and the gate still said no, because 1,292 is not zero. It ended as not accepted after an hour and ten minutes, 40,567,622 tokens and $9.92.

That run is why the gate is progressive. The fair question for one slice is not whether the project is clean. It is whether the slice left it worse. So a baseline is taken before the build, and lint and the type check pass when no new diagnostic appears. A diagnostic's identity is its file, its rule or error code and its message, without the line number, so moving code does not look like a new error. The gate compares identities and not totals, because, in the words of the code, "An equal total with different diagnostics is red." Tests and the build still have to pass outright.

Its last step tripped on formatting, and its reviewer undid a fix. The next slice ran in a lane that also has a reviewer agent. Twice, the phase that commits the plan failed on our commit-message linter. Then the reviewer forced the builder to revert a fix to a test file as "out of scope", and the retest, with the fix gone, failed as you would expect. That run (e5072fa9) cost $2.75 and 24 minutes, and I finished the slice by hand as #675. Two things came out of it: a normaliser that makes any commit header pass the linter, and a decision that scope should be a pattern in the spec and not an agent's judgement.

It ran the wrong spec. Session f25e0286 took four and a half minutes and 430,534 tokens and ended with "nothing to commit — the preceding phases changed no files". It was telling the truth. My shell history had replayed the command for a slice that merged that morning, on another slice's branch. The planner worked that out within half a minute and wrote a verify-only plan, and then the build, test and review phases ran anyway, because a plan had no way to say stop. I told that story from the other side in the post about the lint autofix. Here it matters because my reaction to it shaped the factory. What I asked for that evening was that every run should count "towards real progress, not some dead loops".

Preflight and the plan verdict are the answer. Replaying that same command now stops in about two seconds.

Figure 2. The five failures of the first two days, what each cost where a cost was recorded, and what each one added to the lane in Figure 1.

5. The numbers from 47 runs

The factory keeps a database of its own runs. Between 2026-10-08 and 2026-10-13 it holds 47 sessions, $95.82 and 346,343,301 tokens. It marks 37 as success and 10 as fail.

Fifteen of those are the first two days: demos, questions, and the attempts in section 4. They cost $14.66, and $9.92 of that is the one run.

The other 32 are the strict lane with a contract in the spec, from 2026-10-10 on. They cost $81.17. Everything below is about those 32, because they are the same lane doing comparable work.

Figure 3. Each strict-lane run in order, by cost and outcome. The tall block is the night of 2026-10-11: fourteen runs and $48.70.

Outcomes. 28 ended green and 4 ended red. Of the 28, 21 passed on the first loop and 7 needed one fix loop. No run recovered on a second fix loop.

Time. The shortest run took 9 minutes, the longest 35, and the median 17. Together they took 584 minutes, a little under ten hours of machine time.

Cost. From $0.55 to $5.88 a run, with a median of $2.01.

What the work was.

Kind of slice	Runs	Cost	Ended green
Test files made lint-clean	17	$53.10	15
Loose records given named fields	7	$8.97	6
The last two strictness flags	6	$16.90	5
Two other typed slices	2	$2.19	2

Where the time went.

Figure 4. The 584 minutes by phase. Agents were working for 81% of the time. The deterministic gates took 19%, and the contract, which caught the most for its cost, took 2%.

The builder is where the time and the money go: 55% of the minutes and $58.02 of the $81.17. Planning took 22% of the time and $18.69. All the fix phases together cost $4.47. The contract phase, which runs the pre-commit hook, the scope check and every acceptance command, took twelve minutes across all 32 runs.

What the tokens were. This one surprised me, and it changes how I read a token count.

Figure 5. The 292.6 million tokens of the strict-lane runs. 96.3% were cached context being read again on each step. The models wrote 2.1 million, which is 0.7%.

A run that reports "22 million tokens" did not write 22 million tokens of anything. An agent that takes two hundred steps re-reads its growing context two hundred times, and almost all of that is served from a cache. Of 292,552,132 tokens, 281,777,223 were cache reads. The two agents together wrote 2,102,637. The 32 runs made 3,406 tool calls, 2,351 of them shell commands.

The pattern in cost. Cost followed the number of files a run had to read and edit, not the number of errors it cleared. Across the six flag slices the cost per error ranged from about $0.002 to $0.026, a factor of thirteen, while the cost per file with an error stayed between about $0.02 and $0.06. A slice with many small mechanical fixes in few files is cheap. A slice that has to open many stories and rendered components is not.

What I cannot conclude about models. The builder ran on one model for 12 runs and on another for 20. The median cost was $3.50 for the first twelve and $1.76 for the twenty. I am not going to call that a comparison. The first twelve were all test cleanup, including the largest groups, and the twenty include different and often smaller work. The notes from the night record the one fair observation: the first run after the switch cost "a third of the cost and less than half the tokens of any of the nine runs before it", on a group of only five files. The switch itself happened because of an outage, not an experiment, and nobody ran the same slice on both.

6. What the gates caught, and when the gates were wrong

Section 5 says what the runs took, not whether the gates were right. Eleven of the 32 runs met a red gate at some point. I went through each one, because "the gate fired" and "the gate was right" are different claims.

Six times the gate was right.

• In five test-cleanup runs, the quality gate found a type error that had not been in the baseline. Four were repaired in one fix loop (8d9d178c, 5bd02730, 526ebf58, 84e783fb) and went on to commit. The fifth (732e9e1c) is the outage: the gate caught the error, and then the provider refused the fix phase eighteen seconds in.
• In one run (249f6c5b) the contract's lint check failed three times and the run ended red with nothing committed, at $5.79. Lint really was failing. Section 8 has what was going on, because the cause was the factory's own blind spot.

Five times the gate was wrong and the work was right.

• 529bcf03: the scope check refused a correct build three times. A path in the slice contains [code], a folder name our router uses, and in a file pattern that is a character class. The spec's own list did not match its own file. $0.82 and nine minutes.
• 1ce281e1: a check that compares compiled output before and after called a formatter's line breaks a change.
• aac717ae and f8c0c58d: the first verification failed on a timing test the branch does not touch, because the machine was busy. The second, of the same code, passed.
• 9caf5b89: a check said the rest of the project may not gain a type error, measured by count. The slice fixed one error and the compiler then reported its twin in a file the slice was not allowed to edit. I stopped that run by hand.

Six right, five wrong. I had expected the gates to be the dependable half of this system, and on the days I measured they were right a little more often than not. Three of the five false alarms were faults in checks I had written into specs. A check is code, and code written in a hurry to guard other code has bugs of its own.

Preflight also refused four dispatches before any tokens were spent: once for a stray folder in the checkout, once for a precondition that counted nothing because of a search pattern, and once because I had renamed a branch and not the spec. The fourth was a deliberate replay of the wrong-spec command, to see it stop.

And one fault sat under all of it for days. The factory's code is in a folder Git ignores, but the linter and the type-checker did not ignore it. In the checkout the factory runs in, lint exited with 20 errors and the type check with 37, none of them in a file the repository contains. Because the gate only runs a build when the type check passes, it had never checked a build in the checkout it runs in. Every total I had reported was off by a constant, and the comparison with the baseline hid that, since the same errors were there before and after. That is fixed in #719.

7. What passed every gate and was still wrong

This is the section I would keep if I could keep only one. Section 6 asked whether a red gate was right. This one asks the opposite: when every gate is green, what can still be wrong?

Seventeen runs cleaned up test files: a pilot and sixteen groups. The job was to satisfy new lint rules without changing what any test checks. Fifteen of the seventeen ended green. Every one of those passed the hook, the scope check, the spec's acceptance commands, lint, the type check and the whole test suite.

A test is there to fail when the behaviour it covers changes, so the direct way to check a rewritten test is to change the behaviour. After each run, the tests were reviewed by breaking the code under them. Take a component, change it in a way a user would notice, and run the test file as it was before the run and as it is after. If the old file fails and the new one passes, the run made that test weaker.

Figure 6. Tests found asserting less than before, by cleanup group, as the notes count them, including those found after a group had merged. Groups 6 and 11 ended red and were finished by hand; every other bar is from a run that passed all of its checks.

Of the fifteen runs that ended green, thirteen had left at least one test asserting less. One of the other two had a different fault, a lookup by position that two reviewers flagged. One was clean after 45 breaks. The two runs that ended red were finished by hand, and the trees their builds left had weakened tests as well.

A few of them, so this is not abstract:

• An exact-text assertion became a substring match. The original failed on "try again later, maybe". The new one passed.
• A test that read the text of one specific element became a test that the same word appears somewhere in the card.
• A comparison of two lookups passed when both sides were missing.
• A count of <button> tags became a count by role, which also counts things that only claim to be buttons.
• In one group, eight tests traded a stable hook on the element for a role query, because my own spec's ordering of preferred queries told the builder to.

That last one matters. The agents were mostly doing what they were told. Several of the worst groups followed a rule I had written into the spec generator the day before, to prevent the previous group's problem. One early rule capped how many lint suppressions a run could use, and the planner responded by writing a "suppression budget" into its plan, and tests were weakened to stay inside it. I removed the cap. A rule in a spec is an input to the next run, not the end of a problem.

Other things passed every gate too:

• The contract once checked nothing. In session 8e39a399 the builder ran git commit itself during the build. Every check in the contract compares the tree with the last commit, so each one compared a file with itself, and the scope check saw one changed path where there were fifteen. The run ended green. It was noticed in review because the branch had two commits with the same subject.
• A report said things that were not true. In one run, two statements in the builder's summary were false. One was caught by a break and one by reading the file it was about.
• A check was satisfied the wrong way. To pass the compiled-output check, one run added eleven formatter-ignore comments to five production files.
• A fix phase replaced a commit message. One slice's commit carried a single line about a type error, where the build had written about four thousand characters describing thirteen files.

The time this took a person is recorded for four of the early groups: about 28, 19, 11 and 13 minutes of review after a run of ten to twenty-three minutes. Later reviews used a small script that applies one break and runs the old and new test files against it, and went through 91, 63 and 43 breaks. For those four groups the reading took time of the same order as the run.

The notes hold a sentence about this that I have reread more than once: "Counting assertions measures that a test still asserts, and says nothing about what it asserts on." My gates count things. Whether a test still means what it meant is not a count.

8. Every gate has a run it came from

I said at the top that each gate exists because a named run failed without it. Here is the list, in order. The factory's own test suite grew with it, from 9 tests on the morning of 2026-10-09 to 85 by the end of 2026-10-11.

Figure 7. Tests in the factory's own suite, at each point the notes record. Almost every step is one incident and the test that pins its fix.

What was added	The run behind it	What happened
Progressive gate, compared by identity	4b06828b	$9.92 against a baseline that was already red
Commit-message normaliser	slice 4's runs	the last step failed on formatting, twice
Preflight	f25e0286	the wrong spec on the wrong branch, 430,534 tokens
Plan verdict	f25e0286	the planner knew in half a minute and could not say stop
The contract block in a spec	slice 7's spec	wrong lane named, a target its method could not reach, scope judged by an agent
Project conventions told to every agent	the lint rollout	a recommended lint rule told agents our test cleanup calls were unnecessary, which is false in this repository
Restoring the plan file	5836bbd9	the builder saw the planner's file, judged it "unexpected" and deleted it
Spec generator that measures first	the cancelled autofix slices	three slices were planned around a method nobody had run
A check tries its own pattern first	fe8b82a0	a new check's pattern matched nothing, examined 25 files and passed
A note that the hook rewrote the file	249f6c5b	the builder fixed its file, the hook reformatted it, and nothing said so
A note on a repeated failure	249f6c5b	three attempts on one failure, handed the same words each time
Keeping the fuller commit message	526ebf58	a fix phase's one line replaced the build's full message
Provider stop, with its own exit code	732e9e1c	"no credits remaining" looked exactly like a red gate
Undoing an agent's own commit	8e39a399	the builder committed, and the contract compared each file with itself
Escaped paths in a spec's scope	529bcf03	[code] in a path is a character class in a pattern

Two of these deserve a few more words, because they show the same thing from two sides.

In session 249f6c5b, the builder wrote a suppression comment on the right line. The pre-commit hook then ran the formatter, which re-wrapped the code and left the comment one line away from what it suppressed. The lint check failed. The builder was handed the failure, opened the file it remembered writing, saw the comment where it had put it, and "fixed" it the same way. Three times. The builder was reasoning correctly about a file that no longer existed. Now a failure after the hook says "The file this check read is not the file you last wrote", and a failure identical to the last one says so and asks for a different change.

In session 732e9e1c, the builder's provider returned an error eighteen seconds into a fix phase. The run's summary and exit code were the same as for a red gate, so from the outside it looked as if the code had failed. Now a provider error has its own message, "STOPPED BY THE PROVIDER, not by a gate.", and its own exit code. It deliberately does not pick another account to try: "Which account pays is the operator's decision, not the lane's."

Both fixes are the same idea. When a run stops, the next reader, human or agent, has to be told the true reason, in words, at the place they will look.

9. Where the terminal fits

That reader mostly worked in a terminal. I had planned a post comparing the two ways of working: prompting an assistant in a terminal, and running a factory. I cannot write that post honestly, because no task in these six days was done both ways. There is no baseline. What I can do is describe how the work divided, with the counts I have.

The factory built slices. Everything around the slices happened in a terminal session with an AI assistant: measuring what was there before a spec was written, writing the specs and their contracts, reading each run's diff, breaking components to test the tests, answering the code reviewers, and repairing the gates in section 8.

Figure 8. The two halves, in the two units both sides record. The terminal sessions' transcripts hold no cost, so there is no dollar bar to draw.

	The factory's 32 strict runs	Four terminal sessions
Time span	584 minutes of runs	2026-10-08 to 2026-10-13, on and off
Text the models wrote	2,102,637 tokens	4,311,792 tokens
Tool calls	3,406	3,333
Cost recorded	$81.17	none recorded

So the supervising half produced about twice as much model-written text as the building half, in roughly the same number of tool calls. I do not know what it cost, because those transcripts record tokens and not money, and I will not guess.

Two things I can say from the record, without a comparison:

• The factory's value was in repetition. Seventeen cleanup runs, seven typed-record runs and six flag runs each followed a spec generated from a measurement, with the same checks. I could not have held that discipline by hand across hundreds of files, and the assistant in the terminal was busy doing something else: checking the result.
• The terminal's value was in doubt. Everything in section 7 was found by something outside the factory: a person reading a diff, a script breaking a component, or a code-review bot on the pull request. None of it was found by a gate.

The terminal still wins for a one-off question, a search, or a measurement. I would not start a six-phase pipeline to ask where authentication is handled, although the factory's second-ever run was a question of that kind, for 3.6 cents.

10. What is not built, and what I would not claim

Not built. I have designs and no code for a few things I asked for during the week: a report at the end of every run with its checks, next steps and notes; a way for a run to relay its result to my other sessions; a shelf for ideas that come up mid-run; and a phase where a person approves before the build. There is no preflight that asks the provider whether the account can pay, which is the check that would have saved the outage run. And the one lane with a reviewer agent has no contract phase.

Known holes. The guardrails document lists these as open, in its own words:

• "Nothing reads a plan for a planned change of subject". Several weakened tests were written into the plan first, and nothing checks a plan before it is built.
• "The contract's checks compare with HEAD and not with a named commit". That is how a builder's own commit blinded the contract.
• "Verification is not run alone by anything". Two runs spent a fix loop on a timing test that failed only because the machine was busy.

What I would not claim.

• That the factory saved time. I did not measure the same work without it.
• That one model is cheaper than the other. The tasks differed.
• That the rollback net has caught a real out-of-scope write. In these six days the only breach it reported was the bytecode false alarm. It may be doing its job by deterring nothing, and I have no run that shows it stopping something.
• That a green run is safe to merge unread. Thirteen of fifteen says otherwise.

11. Closing the loop

I started with a sentence from the factory's own code: agents are for the parts that need reading and deciding, and a known command belongs in code. After 47 runs I think that split is right, and I had the boundary in the wrong place.

I had filed "did the tests survive the cleanup" under known commands. Count the assertions, count the test cases, run the suite. All three are commands, all three passed in fifteen runs, and thirteen of those had still lost something. Whether a test means what it meant is a judgement, and the only thing that settled it was changing the code and watching which version of the test noticed.

Seen whole, it is a loop and not a line. A spec goes in, a commit comes out, a reader breaks things, and what the reader finds goes back in as a gate or as a rule in the next spec. Section 7 showed a rule doing harm on its way back in. So any change to the factory gets two questions: what will the next run do with it, and how could a run pass it without doing the work?

So the factory I have now is a gated pipeline that produces a commit I still have to read, plus a growing set of tools for reading it faster. It cost $95.82 and it did hundreds of files of work I did not want to do by hand. It also needed a second half, outside itself, that doubts it. I would build it again, and I would budget for the second half from the first day.

Key takeaways

1. Let code decide and agents propose. Every phase in the lane starts as failed, and no agent grades its own work.
2. Build each gate from a failure you can name. Every gate in the lane was. The checks I wrote into specs from imagination were the ones with bugs.
3. A gate on a codebase that is already red has to be relative. Compare with a baseline, by identity, or one honest run can cost $9.92 and be refused.
4. Rehearse the last step first. The pre-commit hook runs before the slow gate, so a tree that cannot be committed is found in seconds.
5. Check your checks. Five of eleven red gates were wrong. A new check should be made to fail on the thing it exists to catch before it is trusted.
6. Read a token count as re-reading. 96% of the tokens were cached context. The models wrote under 1%.
7. Cost follows files, not errors.
8. Tell the next reader why it stopped. An outage, a reformatted file and a repeated failure each looked like something else until the lane said so in words.
9. A green run still needs a reader. Break the code under a rewritten test and see which version notices.
10. A rule you add is an input to the next run. A cap on suppressions became a budget, and tests were weakened to fit inside it.

What I'd do next

• Write the base commit into each contract check, so a builder's commit cannot make a check compare a file with itself.
• Have something read the plan for a planned change of what a test looks at, before the build spends anything.
• Run verification alone, so a busy machine cannot fail a timing test and start a fix loop with nothing to fix.
• Ask the provider before the run whether the account can pay.
• Make the break review a phase, with a standing break for every new role query a run adds.
• Send the local fixes upstream to the project it began as, starting with the bytecode pattern and the progressive gate.

Evidence

All times UTC. Session ids are the factory's own eight-character run ids.

• Totals: the factory's database, table sessions, read on 2026-10-13: 47 rows, total cost $95.8240, total tokens 346,343,301, status success 37 and fail 10. The 32 strict-lane runs of a spec with a contract add up to $81.17 and 292,552,132 tokens; the other 15 to $14.66. Each run's closing log line agrees with the database to half a cent, except session 4b06828b, whose log closes at zero because its id was used again for a re-verification (the database has $9.9178 and 40,567,622 tokens), and two sessions with no folder on disk (55b868df, 2d6316a7).
• Per-run measurements: every phase's start and end, each agent call's model, tokens and cost, and every tool call, read from events.jsonl for each session into data/factory-runs.json in the release's running notes. Outcomes: 28 success and 4 fail (249f6c5b, 732e9e1c, 529bcf03, 9caf5b89); 21 with no fix phase and 7 with one. Wall time 9.0 to 35.0 minutes, median 16.9, total 584. Cost $0.55 to $5.88, median $2.01.
• Time and cost by phase: build 321 minutes, plan 129, verify 74, fix 23, baseline 22, contract 12, commit 2. Agent cost by phase: build $58.02, plan $18.69, fix $4.47.
• Tokens: cache reads 281,777,223 of 292,552,132 (143,369,216 and 102,933,006 for the builder on its two models, 35,475,001 for the planner); output 2,102,637 (516,625, 768,079 and 817,933). Tool calls 3,406, of which bash 2,351, read 396, edit 347.
• By kind of slice: 17 test-lint runs $53.10, 7 typed-records runs $8.97, 6 strictness-flag runs $16.90, 2 others $2.19, from each spec's title. By day: 2 runs and $2.36 on 2026-10-10, 14 and $48.70 on 10-11, 8 and $11.01 on 10-12, 8 and $19.09 on 10-13.
• Builder model: openai/gpt-5.3-codex in the 12 runs from 5836bbd9 to 732e9e1c, anthropic/claude-sonnet-5-5 in the 20 from 435f786b on, from each agent call's own record. The switch is a commented-out line in the configuration, dated 2026-10-11, and the daily record's entry at 15:49.
• The lane and its gates: adw_plan_build_test_quality.py (the phase summary, MAX_FIX_LOOPS = 3, the commit condition), preflight.py (the eight checks and the refusal message), contract.py (the four fields and their reasons, the hook rehearsal, the two notes), quality.py and quality_diagnostics.py (the four commands, the two modes, what an identity is), permissions.py (the snapshot and rollback, and the comment about git checkout), verdict.py, commit_message.py, git_helper.py, provider_stop.py, and ogsf.config.yaml (the agents, their purposes, tools, write permissions and the protected files).
• The first two days: incidents.md for the import error, the fourteen .pyc files and session 4b06828b (1,735 to 1,292, three verify phases, not accepted); metrics.md for $9.92, 1h10m and 40,567,622 tokens, and for e5072fa9 ($2.7549, 24m22s) and f25e0286 ($0.3362, 4m37s, 430,534 tokens); the guardrails document for the commit linter failures and the reviewer's revert.
• Red gates: swimlanes.md for the new type-error identities in 8d9d178c, 5bd02730, 526ebf58, 84e783fb and 732e9e1c, and for 249f6c5b's three contract failures; incidents.md for 529bcf03's scope ("27 changed paths against 26 patterns"), the timing test in aac717ae, and the lint and type-check errors from the ignored folder (20 and 37); the daily records for 1ce281e1, f8c0c58d and 9caf5b89.
• Tests left asserting less: the row of that name in metrics.md, group by group, with how each was found; incidents.md for the suppression budget in fe8b82a0, the builder's commit in 8e39a399, and the hook traded for a role in 8cd78a4c. Review times and break counts are in the same file's row "Human review after a green run".
• The factory's own tests: 9 at the start of 2026-10-09 and 85 on 2026-10-11, with the steps between, from the daily records; 85 confirmed on 2026-10-13 by counting test functions in adws/tests.
• The terminal sessions: four transcripts for this repository, 2026-10-08T17:30Z to 2026-10-13T13:19Z: 3,247 replies, 4,311,792 output tokens and 3,333 tool calls, each counted once by its id. The transcripts record no cost.
• Where the factory comes from: disler/super-simple-software-factory, MIT licence, copyright 2026 IndyDevDan; my copy keeps that licence and notice, and its README calls it a customized derivative. The two sentences quoted in section 1 are in the upstream code, by a search of that repository on 2026-10-13; the same search finds no preflight there. The rows of section 8's table are the local additions, from the daily records.
• Why it was built, and where the name comes from: the two reasons are mine, stated on 2026-10-13. The history is from Aaen, Bøttcher and Mathiassen, "The Software Factory: Contributions and Illusions" (Proceedings of the Twentieth Information Systems Research Seminar in Scandinavia, Oslo, 1997), which cites Cusumano, "The Software Factory: A Historical Interpretation" (IEEE Software, 1989), for Bemer, McIlroy, Hitachi, System Development Corporation, NEC, Toshiba and Fujitsu; I read the survey, not Cusumano's paper. Greenfield and Short's use of the name is from their book's own site. StrongDM's rules, its spending yardstick and the Dark Factory label are as quoted in Simon Willison's post of 2026-02-07; I have not seen StrongDM's factory or verified its results. All read on 2026-10-13.
• Not found, and left out: what the factory cost to build, any run in which the rollback undid a real out-of-scope write, and a date for when the template's placeholder quality commands were replaced with real ones.

Nothing above contains a credential, a token or a player's identity.