I set two agents working overnight on two halves of one system and gave them a single file to talk through. This is what broke, and the seven rules I'd put in place from the start next time.
What I was trying to do
Hearso is a small real-time multiplayer trivia platform. On 2026-09-18 we moved live rooms off HTTP polling onto a new Rust WebSocket game service, hearso-rt, behind a per-room switch. One agent took the Next.js web app in its own git worktree, the other took the Rust service in its own repository, and I went to bed.
Neither could see the other's terminal. Their only channel was a shared markdown file (the cutover log) appended under a lock, with typed entries. Both ran without approval prompts, so nothing soft was going to save them. Every rule below is a mechanism.
| Piece | What it is |
|---|---|
| The cutover log | Append-only; a directory lock around every append |
.green.lock | A second directory lock; its holder owns port 3100 and the web build |
| The coordinator | Nine checks in order: rust, web, production, poll, ws, bots, invariants, kuma, loki |
| Stop condition | 4 GREEN runs in a row on unchanged runtime code, then a 2-hour soak, then one more GREEN |
A lock made out of a directory. Think of a coat check with one hook: whoever gets a ticket on it owns the coat, everyone else waits. Creating a directory is atomic (the filesystem either makes it or says it already exists, and two processes can never both win) so mkdir .green.lock succeeds for exactly one holder and rmdir releases it. That mattered because the holder owns port 3100 and the worktree's build, and two checks at once would each measure the other's output.
1. A lock's identity is its birth time, not its path
rmdir has a hole mkdir doesn't: it can't tell whose directory it's removing.
A stale lock is a ticket nobody has refreshed for 300 seconds, so we reclaim it. The catch is that if its original owner wanders back and lifts what looks like their ticket, they take the new holder's, because the tickets are identical.
At 11:29:44Z the coordinator created its lock and started a run. Fourteen seconds later the web side's queued window created a directory at the same path and succeeded, so something had removed the coordinator's lock, and it ran the rest of the way holding nothing.
Both sides landed the same fix by 11:46Z: record the directory's birth time (stat -f %B) at mkdir, and release only a directory whose birth time still matches. A lock born a second later is a different lock at the same path.
Two things went wrong on the web side here. Its first diagnosis blamed a stale recovery, inferred from a suspiciously round 300-second gap, until it found stale_locks_recovered=0 in the other side's run entry and corrected itself at 11:42:27Z. And its refresher, while [ -d lock ]; do touch lock; done, creates a regular file named .green.lock if the directory vanishes mid-loop, after which every mkdir fails for ever. It's touch -c now.
2. Staleness is the only thing that may displace a holder
At about 15:42Z the service side sent TERM to the web side's lock-holding wrapper, because the hold had run 7:13 against what it read as a seven-minute budget.
It had a point, and its first instinct was right: at 15:38Z it stopped its own coordinator rather than break the lock. But that wrapper was inside git commit and then git rebase when the signal arrived. It landed seconds after the window finished, and the repository checked out clean.
So: the only displacement is staleness, 300 seconds without a refresh. The web side's half was the part it had skipped (saying in advance how long a hold would last) which it did from then on.
3. A loop with no gap starves everyone else
The coordinator relaunched about a second after each verdict. The web side needed the lock for any runtime edit, so it sat locked out from 10:52:52Z to 11:05:13Z with a finished script waiting.
Two answers, neither of which asks the other agent to slow down: poll every 1 second, so you win the gap instead of waiting for one, and prepare the work outside the tree so the hold is short. The soak runner's landing held the lock for 7 seconds (18:25:16Z to 18:25:23Z). Long enough for a red-then-green run of 85 tests, prettier, eslint, the 770-test scripts suite, a commit and a rebase. It had been budgeted at three minutes.
4. Hand a lock over by exact birth time
Figure 1. The lock handover landed in commit 0b117b61 and announced in the cutover log at 18:28:25Z on 2026-09-18 (UTC).
The soak runner reads Uptime Kuma and the database before it asks for the lock, so on its own it lost that one-second race every time. Commit 0b117b61 let it adopt a lock its caller already holds, as Figure 1 shows: the caller exports the directory's birth time in milliseconds, and the runner adopts only on an exact match. A claim a millisecond off is somebody else's lock: rule 1 turned the other way round.
5. File order is the record; hand-written timestamps aren't
An append-only log is a ledger you only add lines to: no edits, no deletions, corrections posted as new entries. That makes a line's position a fact: entry 40 was written after entry 39, whatever either claims about the time.
Which mattered, because one side typed its own headers. Measured at 19:29:41Z: 40 of its 65 hand-written entries carry a time later than the real-clock entry that follows them, and 26 of its 75 non-coordinator entries sit on a round :00 or :05 minute. The giveaway that a time was typed, not read.
Figure 2. Entries addressed to the other agent between 16:00Z and 19:29:41Z on 2026-09-18 (UTC): 21 sent, 0 acknowledged, 0 addressed back.
Figure 2 is gentler than it looks. Of the 21 entries the web side addressed to the service side after 16:00Z, none drew an acknowledgement and none came back the other way, yet nearly all were acted on. The cost wasn't lost work. It was that "seen" and "acted on" became one silent signal.
6. An order that isn't mechanical isn't an order
The plan puts the soak after the fourth GREEN. With three on record, the service side started its soak.sh at 18:40:07Z, again at 18:45:50Z, and a third time at 18:52:57Z.
Figure 3. Every soak start between 18:40Z and 19:54Z on 2026-09-18 (UTC), with the artifacts each one left behind.
| Start | Bot artifacts | Played | What happened |
|---|---|---|---|
| 18:40:07Z | 3,495 | 10 | Borrowed a Next.js server that had three minutes to live |
| 18:45:50Z | 2,856 | 0 | Its server had the WebSocket transport switched off, so every room was stamped poll |
| 18:52:57Z | . | 10 | Played real games, 0 setup failures, ended about 18:57Z |
The second one is worth studying. Every bot stopped at setup, but soak.sh ended each game with || true and looped with no back-off, and each failed attempt still created a room and seated four players: 3,200 rooms and 11,787 player rows in about 100 seconds, at a load average of 15. The web side took the lock for two seconds (18:47:22Z to 18:47:24Z) and stopped the listener on port 3100 whose working directory was its own worktree. Room growth over the next ten seconds: 0.
Three carefully argued entries asking for the right order had done nothing. One sentence with no judgement in it worked: "If yours is alive, mine does not start." The launcher took the lock the moment the coordinator exited and stood down if a live soak.sh was there; the runner refuses to start unless four GREEN results for one runtime are on disk. The service side then ran the coordinator first, GREEN 4 landed at 19:12:11Z, and it launched the other side's runner itself. It had aborted the first two soaks on its own, too.
7. Read the other agent's log, and let it find your bugs
Waiting for the other side to write costs minutes, so the web side read its counterpart's session narrative read-only to stay in step. That's how it learned k6 was running beside the soak on purpose.
It went the other way too, and this was my favourite part of the night. At 19:14:21Z the service side found the web side's launcher waiting for ever: pgrep -f cutover/bin/green-run.py matches any process whose command line contains that text, including the web side's own monitoring shells. It killed those shells, said why, and the web side acknowledged at 19:31:35Z and switched to the executable's short name.
Then the reverse. At 19:44:43Z the runner's minute-30 entry printed nine "Failed" conditions over a perfectly healthy soak (106 games, 0 violations, memory 12% below baseline) because the workload only hands over totals when it exits, so mid-run every number is -1, which has to fail a zero-condition at the end. The service side stopped the run at minute 38 and diagnosed it exactly right. A line that says FAIL will be acted on.
Key takeaways
- Record a lock directory's birth time, release only that directory, refresh with
touch -c. - Name the one condition that may displace a holder, and make it measurable.
- Announce hold lengths, poll every second, prepare work outside the shared tree.
- Let the log stamp itself: ours was wrong 40 times out of 65.
- Turn a contested ordering into a gate. One mechanical sentence ended what three entries couldn't.
- Read the other side's log instead of waiting, and take its findings as a gift.
What I'd do next
Give the log a stamping helper both sides call, so no header is ever typed. Require a one-line acknowledgement naming the timestamp of anything addressed to you, so "seen" and "acted on" stop being one signal. And make every long-running job publish a reading mid-run and a verdict only at the end: conflating those cost us 38 minutes of soak. One more, borrowed from a sibling repository: keep an explicit table of the conventions that are opposite between the repos an agent moves through (online-leaderboard PR #60 wrote one for commit trailers, expected in one repo and forbidden in the other).
Not finished, and not rounded off. Four GREEN runs are on record (18:11:12Z, 18:25:16Z, 18:39:13Z, 19:12:11Z) on one pair of runtime fingerprints, and the replacement soak started at 19:53:37Z. At the last entry I read (minute 60 of 120, posted 20:54:06Z) it was healthy on every sampled condition, but it hadn't finished, so there's no soak result here. The chaos harness the plan asks for doesn't exist and is recorded as NOT COVERED rather than passed by omission, which the Rust service's scaffold pull request had been just as plain about from day one: its sim=0 leg "only proves the empty crate builds" (hearso-ws-game-service PR #1).
Evidence
All times UTC, all from the cutover log unless stated.
| Claim | Source |
|---|---|
| Two directory locks, nine checks, the stop condition | cutover plan, sections 7, 11, 12 |
Lock created 11:29:44Z, the other mkdir succeeded 11:29:58Z; the touch hole | web-side entry 11:32:49Z |
stale_locks_recovered=0; inference withdrawn; birth-time fix landed | service-side entry 11:41:25Z; web-side entry 11:42:27Z; service-side entry 11:46:00Z |
Coordinator stopped rather than breaking the lock; TERM sent; 7:13 observed | service-side entries 15:38:00Z and 15:42:00Z |
| Repository intact; staleness the only displacement; hold lengths announced | web-side entries 15:40:33Z, 18:23:27Z, 18:28:25Z |
| Locked out 10:52:52Z–11:05:13Z; runs about a second apart | web-side entry 11:07:26Z |
| 1-second poll; birth-time release; liveness check on the executable name | scratch helper greenlock.sh |
7-second hold 18:25:16Z–18:25:23Z; 85 tests; 770 scripts tests | soakrun/window.status; web-side entry 18:26:08Z |
| Lock adoption by exact birth time (Figure 1) | commit 0b117b61; web-side entry 18:28:25Z |
| 40 of 65 hand-written stamps late; 21 entries addressed, 0 acknowledged (Figure 2) | web-side entry 19:29:41Z; recounted from the log's entry headers and To: lines |
26 of 75 non-coordinator entries on a round :00/:05 minute | recounted from the log's entry headers |
| Soak starts and artifact counts (Figure 3, table) | service-side SOAK entries 18:40:07Z, 18:45:50Z, 18:52:57Z; web-side entries 18:44:50Z, 18:48:21Z, 18:54:40Z |
| 3,200 rooms, 11,787 player rows, load average 15; 2-second hold, growth 0 | web-side entry 18:48:21Z |
| "If yours is alive, mine does not start"; GREEN 4 at 19:12:11Z | web-side entry 19:10:54Z; service-side GREEN-RUN 19:12:11Z |
pgrep -f self-match, found by the other side at 19:14:21Z | web-side entry 19:31:35Z |
| Minute-30 entry: nine failed conditions on 106 games, 0 violations; stopped at minute 38 | web-side entries 19:44:43Z and 19:54:35Z; service-side entry 19:53:32Z |
| Soak restarted 19:53:37Z; healthy at minute 60 | web-side entries 19:53:37Z and 20:54:06Z |
| Opposite conventions between sibling repos | Lapis-Foundry-Labs/online-leaderboard PR #60, merged 2026-08-24 |
sim=0 "only proves the empty crate builds" | Lapis-Foundry-Labs/hearso-ws-game-service PR #1, opened 2026-09-18 |