A soak of our new WebSocket game service passed on the fifth attempt. The four before it failed on sizing, on ordering, and once on a message our own tool wrote.
What I wanted to know
I wanted to know whether hearso-rt, our new Rust WebSocket game service, can hold rooms open for two hours without losing a game, a connection or a database row. We used a purpose-built soak runner driving ten rooms of bot players through Next.js and the socket, sampling Uptime Kuma and Postgres beside it.
Hearso is a small real-time multiplayer trivia platform: a Next.js web app, hearso-rt, local Postgres and Valkey. On 2026-09-18 a web-side agent and a service-side agent ran an overnight cutover moving live rooms onto hearso-rt behind a per-room switch, locally, nothing deployed. They worked through one append-only file, the cutover log, and every number here comes from it.
What a soak test is
Think of a tap that does not drip. Turn it on for a second and it looks fine; leave it running overnight and you find out whether the washer holds. A soak test runs a normal workload at a normal rate for a long, fixed time, watching for faults that only show with age: memory that never comes back, handles that pile up, a queue falling behind.
Ours was two hours of bot games in ten rooms, with one number aimed at that: hearso-rt's resident memory had to grow less than 10% between minute 15 and minute 120. Minute 15 is the baseline because a process that has just started is still filling caches.
hearso-rt holds a socket per player for a whole game, and every live room is state it owns until something cleans it up. Ten seconds of traffic cannot tell it from one that never tidies up.
Seven conditions, fourteen numbers
Section 12 of the plan set the bar: zero invariant violations, zero 1011 closes, zero 5xx from Next.js, no outbox row unprocessed for over 60 seconds, every Uptime Kuma monitor up throughout, memory growth under 10% from minute 15, and p95 snapshot delivery under 50 ms.
Two need a word. A 1011 is the close code a WebSocket server sends on an internal error it cannot recover from. An outbox row is the record we write in the same transaction as a finished game, and one unprocessed after 60 seconds means delivery is falling behind while the games look fine.
The runner turns those seven into fourteen numbers. The other seven are mine: one fault the plan did not name, two asking whether the workload ran at all, one enforcing the rule that a runtime commit voids the soak, and three asking whether the run was measured.
That last group exists because of a trap we have hit before. A maximum over an empty set of samples is zero, and zero violations is what a passing soak looks like. So the runner never reports a number it did not take: it reports -1, a sentinel, a value outside the real range standing for "no value". Ours read -1 for invariant violations at minute 30 and 0 at minute 120. At the end, -1 fails: a condition nobody measured is not one that passed.
That rule is right, and it is what stopped a healthy run 38 minutes in.
The gates
Figure 1. The order the runner enforces, from scripts/cutover/soak-run.mjs.
The runner refuses to start unless three things hold, checked before it locks or launches anything. Four GREEN verification runs must end the results list on identical runtime fingerprints. Ours were 18:11:12Z, 18:25:16Z, 18:39:13Z and 19:12:11Z, nine checks passing each time on one Rust commit. Then /health must answer 200, and every sampler must return a reading at minute 0.
I would not give up that third gate. Earlier that evening, Kuma credentials that never reached a checker cost us a red run and a quarter of an hour; asked at minute 0 the same fault costs a minute. Kuma listing zero monitors is not a reading either: zero is what a login page parses to.
Only then does it take the lock that owns port 3100, start Next.js with the WebSocket transport switched on, and start the bots.
The workload
The bots play ten rooms at once, four to eight seats each, joining through the same Next.js ticket routes a browser uses. A room plays three games and is abandoned, a new one opening in its slot: 121 in all by minute 118.
The chaos is seeded, so a bad night can be replayed. On each snapshot a bot receives during a game, it reconnects with probability 0.003 and leaves for good with 0.001, and only leaves while more than three seats remain. The host can be the one who leaves, which is the only thing that exercises host succession.
One seat abstains from every rematch vote, so the room must reach a majority carrying a non-voter. Snapshot delivery is timed per frame, the service's own timestamp against the moment this process received it.
Three false starts and a rehearsal that "failed"
Figure 2. Eleven events on 2026-09-18 (UTC), from the cutover log.
The soak script that arrived first could not fail. For two hours it ran two checkers every thirty minutes and slept, starting no room, no bot and no socket, ending every command in || true. Sixteen lines, and a two-hour soak of an idle service passes by construction: the shape of a sweep job of ours that ran for a day and collected zero of twelve qualifying rooms (hearso-web #425).
Three soaks were then started before the fourth GREEN run existed. The first, 18:40:07Z, borrowed a Next.js server my three-minute rehearsal was about to stop: 3,495 bot evidence files, 10 played and 3,485 failed at setup. The second, 18:45:50Z, ran against a Next.js restarted with the WebSocket transport switched off, so every room was stamped poll and the ticket route answered 409.
No loop checked an exit code and none paused, so ten ran flat out: 2,856 artifacts, every one a setup failure, and 3,200 rooms with 11,787 player rows written in about 100 seconds at load average 15. The third, 18:52:57Z, was a real workload but still one GREEN early.
My own rehearsal "failed" too: games_finished 0, workload_exit 1, rss_growth_percent 45.98, none of it about the service. A game takes about 200 seconds and I had sized the rehearsal at 180; and the memory baseline is an eighth of the run, so three minutes measures growth from minute 0.
What it did prove is the plumbing: nine samples at twenty seconds, every sampler reading every time, Kuma 7 of 7, zero violations, zero 1011, zero 5xx, p95 1 ms, and the runner releasing its lock by itself.
The 38 minutes
Figure 3. The bug: a progress entry running the end-of-run verdict over a run in progress.
The fourth GREEN landed at 19:12:11Z and the soak started at 19:14:31Z. At minute 14 the log records 35 games finished, 68 reconnects, 17 leaves, 2,550 commands accepted, zero of every fault counter, p95 1 ms, 14 of 14 samples taken.
At 19:44:43Z the runner posted its half-hourly progress entry and listed nine of the fourteen conditions as failed: invariant violations, 1011 closes, unexpected closes, 5xx, snapshot p95, games finished, workload exit, minutes run, sample coverage. The service-side agent read it, worked out why, and stopped the run at minute 38. Its reading then: 106 games, 184 reconnects, 43 leaves, zero of every fault counter, p95 1 ms, memory 11.69% below its minute-15 baseline.
That was my bug, in my entry. The workload hands its totals over only when it exits, so mid-run every workload number is -1, and -1 fails a zero-condition exactly as it must at the end. minutes_run and sample_coverage_percent cannot pass early either. I had run the end-of-run verdict over a run in progress, reviewed that function, tested its titles, and never once read a mid-run entry the way a stranger would.
A line that says FAIL will be acted on. Nothing about the decision to stop was unreasonable.
While the restart ran I posted a companion entry beside each progress entry, with the live counters and a rule naming the five numbers truly sampled each minute. The real fix is prepared and not yet landed: progress entries carry a reading instead of a verdict, end-only conditions read "pending", FAIL does not appear before the end, and signals release the lock. The verdict function is byte for byte unchanged: 153 tests where there were 94, and 14 mutants with none surviving.
The results
The restart ran from 19:53:37Z to 21:53:43Z and passed every condition the runner measures.
| Condition | Value | Limit | Verdict |
|---|---|---|---|
invariant_violations | 0 | 0 | pass |
closes_1011 | 0 | 0 | pass |
unexpected_closes | 0 | 0 | pass |
http_5xx | 0 | 0 | pass |
outbox_pending_over_60s | 0 | 0 | pass |
kuma_samples_not_all_up | 0 | 0 of 120 | pass |
rss_growth_percent | 1.95 | under 10 | pass |
snapshot_p95_ms | 1 | under 50 | pass |
games_finished | 350 | above 0 | pass |
workload_exit | 0 | 0 | pass |
runtime_changed | 0 | 0 | pass |
minutes_run | 120 | 120 | pass |
sample_coverage_percent | 100 | at least 90 | pass |
db_samples_missing | 0 | 0 | pass |
Figure 4. Workload totals, 2026-09-18 (UTC), from the workload's evidence file.
The bots finished 350 games, restarted 236 as rematches, reconnected 553 times, left 177 times, changed settings 70 times, and had 19,926 commands accepted against 240 refused. A refusal is an answer, not a fault: a bot racing a timer is told 409.
Figure 5. Cumulative counters, 11 readings per series: six from the cutover log, five from a recorder that started at minute 71.
Both grow close to a straight line: 42 games at minute 16, 82 at 30, 170 at 60, 260 at 90, 350 at 120. A service degrading with age bends that curve down.
Figure 6. hearso-rt RSS, 24 points, each the mean of the four 30-second samples in a 2-minute bucket (96 samples, minutes 71 to 118). Minutes 0 to 70 are not covered.
Memory wandered rather than climbed: growth against the minute-15 baseline read 0.45% at minute 30, 1.89% at 60, -1.56% at 90 and 1.95% at the end, against a 10% limit. Over minutes 71 to 118 the recorder saw resident memory between 25.0 and 29.8 MB on one unchanging process.
Snapshot p95 read 1 ms in every sample, from 192,574 timed frames by minute 118. All of it is loopback on one laptop, so that says the service is not the bottleneck and nothing about a real network. hearso-rt's CPU held a median of 17.6%.
What this does not prove
The plan asked for a chaos harness running continuously beside the soak, and there isn't one. The crate meant to hold it is a 43-line arithmetic loop that touches no room engine; looping it beside the soak added load and injected no fault. The seeded reconnects, leaves and host departures stood in against the real service, which is worth something and is not the same thing.
k6 ran beside the soak at one virtual user in 30-second iterations, but its totals were never posted, so I am not claiming it. A process being alive is not a result.
Ten rooms on one laptop is a correctness-over-time test, not a capacity test, and nothing is deployed. Two hours is a weak leak test too: a few kilobytes per room would sit inside the noise we measured. The run also used one seed, which is a replayable path, not a survey.
Key takeaways
- A reading taken mid-run and a verdict taken at the end are different questions. One function doing both is how a healthy run got stopped.
- Keep the sentinel for "not measured", and keep it failing at the end. It guards the worse bug, where a maximum over zero samples is zero and zero is what passing looks like: the shape of hearso-web #106, where a scanner's "incomplete" state read as green.
- Make "was this measured at all" a condition of its own. Sample coverage and missing readings caught nothing here, which is how we know they were watching.
- Put the gates before the lock: refusing at minute 0 costs a minute, refusing at minute 120 costs the night.
|| truearound a workload means the run cannot fail, and a loop with no back-off turns that into 3,200 junk rows in 100 seconds.- Write for whoever reads the entry at three in the morning without the code in front of them. They will act on the word FAIL.
What I'd do next
- Land the prepared fix, then read one of its progress entries cold before trusting it.
- Build the chaos harness the plan asked for: faults injected into the room engine and checked against the invariants, not an arithmetic loop beside it.
- Record k6's command, rate and totals, so it can be reported as covered rather than seen.
- Sample memory every 30 seconds from minute 0 and keep the file; the recorder behind Figure 6 only started at minute 71.
- Repeat on two more seeds, for longer, with more rooms, on a quieter machine, and only then say anything about capacity.
Evidence
All times UTC, all 2026-09-18; the front matter lists every entry consulted. hearso-web #425 and #106 are in Lapis-Foundry-Labs/ice-breaker-trivia-game.
- The 14 final values (table, Figures 1 and 3): the runner's END entry, web-side 21:53:43Z, and
var/soak-authoritative-2.json(examined: 120,conditions_failed: 0,rss_growth_percent: 1.9476905954368393). - Workload totals (Figure 4):
soak-bots.json. 350 Games, 236 rematch restarts, 553 reconnects, 177 leaves, 19,926 commands ok, 240 refused, zero on every fault counter, p95 1 ms; the 70 settings changes from web-side entry 21:53:50Z. - Readings at minutes 16 to 120 (Figure 5): web-side entries 20:10:09Z, 20:23:55Z, 20:54:06Z, 21:03:03Z, 21:24:25Z, 21:53:50Z.
- Memory, CPU and frame count (Figures 5 and 6): a read-only recorder, 99 samples every 30 s from 21:05:23Z to 21:54:45Z. 96 Inside the soak (minutes 71 to 118), three after the runner exited, one hearso-rt process throughout. Its minute-90 reading is identical to the log's.
- Gates, sentinel, limits, seats, chance and the prepared fix (Figures 1 and 3):
scripts/cutover/soak-run.mjsandsoak-bots.mjs, commitsd700c5ff(85 tests, 29 mutants) and8bbd915f; the unlanded fix has 153 tests where there were 94, 14 mutants, none surviving, and leavessoakRunVerdictunchanged. - The night's sequence, and what was not covered (Figure 2): service-side entries 18:40:07Z, 18:45:50Z, 18:52:57Z, 19:53:32Z; web-side entries 17:52:11Z, 18:42:25Z, 18:44:10Z, 18:48:21Z, 19:14:31Z, 19:44:43Z, 19:53:37Z, 19:54:35Z, 22:08:52Z.
No credential, ticket, cookie or player identifier appears in anything quoted here; all of it is counts.