I wanted to know whether the small scripts policing an overnight cutover were genuinely tested or just accompanied by tests, so we changed one line of each on purpose and watched for something to go red. Here's what that caught, what it cost, and the three places it didn't help at all.
Why I bothered
On 2026-09-18 a web-side agent and a service-side agent moved Hearso's live trivia rooms onto a new Rust WebSocket service overnight. A handful of small scripts decided whether that was going well: a health checker, a bot workload, a database invariants reader, and a runner judging a two-hour soak against seven numbered conditions.
Those scripts were the only thing between "the system is fine" and "nobody measured". One had already taught us that: an Uptime Kuma checker that wrote examined: 0 and exit 1 whether Kuma was down, unreachable, or missing its credentials. Five coordinator runs were lost before we could tell those apart. So every new test got one extra step by hand before it landed.
The habit, stated plainly
A smoke alarm that's never gone off isn't a working smoke alarm, it's an object on a ceiling. You hold something smoky under it and see what happens.
Precisely: a mutant is one deliberate change to the code under test. An operator flipped, a guard deleted, a constant moved by one. Run the suite. If a test goes red the mutant is killed and that test is proven to hold that behaviour; if everything stays green the mutant survived, and it has named a behaviour nothing checks.
In our Kuma checker, up = values.count(1) became up = values.count(1) + values.count(2), so pending counted as up. One test went red, so that mutant died and the behaviour is pinned. It mattered because these scripts decided whether a two-hour soak passed, and a green nobody can prove wrong isn't evidence.
Figure 1. The loop we ran by hand on every new test. Of 51 mutants across three scripts on 2026-09-18 (UTC), 4 took the lower branch.
We picked every mutant by hand rather than generating them. These were files of a few hundred lines, and the mutants worth running were the ones you could name in a sentence.
One example, end to end
The replacement Kuma checker reads Kuma's Prometheus endpoint over plain HTTP and should be green only when every monitor is UP. Kuma encodes status as 1 UP, 0 DOWN, 2 PENDING, 3 MAINTENANCE.
# the code
up = values.count(1)
return (0 if values and up == len(values) else 1), {"examined": len(values), ...}
# the mutant: PENDING quietly counted as UP
up = values.count(1) + values.count(2)
# the test that kills it
def test_pending_is_not_up(self):
code, ev = check.verdict(ENV, answers(200, row("a", 1) + row("b", 2)))
self.assertEqual((code, ev["metrics"]["pending"]), (1, 1))
One line changed, one test red, mutant dead. The checker has 11 tests and took 8 mutants: pending counted as up, an empty Kuma reported green, missing credentials reported red instead of "not configured", a 401 parsed as a 200, every metric row counted as a monitor, examined floored at 1, the configured Kuma address ignored, http_status dropped. All 8 died.
The count isn't the point. The point is that the checker now gives four different readings for four real failures: real credentials exit 0 with examined 7; a wrong password exit 1 with http_status 401; no credentials exit 125; a URL nobody serves exit 1 with http_status 0.
What it caught
Figure 2. Every mutant run by hand against three tooling scripts on 2026-09-18 (UTC): 51 in total, 4 of which survived their first pass.
| Script | Tests | Mutants | Survived at first | Where |
|---|---|---|---|---|
| Kuma checker, plain HTTP | 11 | 8 | 0 | scratch, offered to the other side |
| Soak runner, first landing | 85 | 29 | 3 | commit d700c5ff |
| Lock adoption | 94 | 1 named | 0 | commit 0b117b61 |
| Progress-entry fix | 153 | 14 | 1 | prepared, not yet landed |
The soak runner's three survivors are the whole argument, because each named a test nobody had written:
-1passing a zero-condition. The runner marks an unmeasured number-1. Letting-1satisfy a "must be zero" condition broke no test, which is a soak passing by never measuring.- Memory from the first sample instead of minute 15. The rule is RSS growth under 10% from minute 15, to skip warm-up. Measuring from minute 0 still produced a number, and everything stayed green.
- No preflight. Removing the refusal to start (before the lock, before four GREEN runs exist, before the service answers) broke nothing.
Each got its test before the file landed. The lock-adoption change had one mutant worth naming and it died: a birth time that's nearly right must not be accepted. The later progress-entry fix had one survivor, also given a test: a title that could say "healthy so far" above a list of named reasons it wasn't.
Figure 3. Tests passing on the soak runner at four points on 2026-09-18 (UTC): 71 as handed over, 85 landed, 94 after lock adoption, 153 in the prepared fix.
Figure 3 is one file growing: 71 tests as a delegated sub-agent handed it over, 85 when it landed after the mutant sweep, 94 with lock adoption, 153 in the fix prepared later. Not all 14 tests between 71 and 85 came from survivors: three did, by name, the rest from the same review pass.
What it cost
Less than it sounds, and I'm not going to invent a number for the part nobody timed. The landing window stamped itself: the 85-test file ran in about 1 second (red 18:25:17Z, green 18:25:18Z) and the whole 770-test scripts suite in about 3 seconds (18:25:18Z to 18:25:21Z). At roughly a second a run, 29 mutants is under a minute of machine time.
The real cost is human: choosing a mutant worth running and writing the test that kills it. That wasn't measured, so it isn't reported. The harness itself was twenty lines of Python: back up the file, apply one anchored replacement, run vitest, restore, diff.
Where it didn't help
Hand-picked mutants aren't coverage. We chose mutants we could name, so we tested the properties we already had in mind. Eight mutants against a 40-line checker is a good afternoon, not a proof the checker is right.
Some mutants can't be killed. An equivalent mutant changes the code without changing its behaviour, so no test can catch it. Ours: on the browser's protocol codec, accepting a JSON list where a plain object is required broke nothing, because a list carries no v claim and is refused a step earlier anyway. The guard was dead code, so we deleted it and retargeted the test at JSON null, which does throw without it. A survivor isn't always a missing test; sometimes it's a line that shouldn't exist.
The thin real-I/O layer stayed untested, and that's where the real bugs were. A test seam is the split between logic you can call with fake inputs and the wiring that touches the world: here, pure exported functions plus a realDeps() object holding the process, network and filesystem calls. The runner's notes admitted it: "realDeps() was never executed. By me or by the tests." Both of the night's real bugs were in there: greenRunAlive() used pgrep -f <path>, which matches any process whose command line merely contains the path (our own monitoring shells included), and a missing signal handler let SIGINT skip the finally and leave the lock directory on disk.
A fully tested function can still be wrong about its job. The runner built its half-hourly progress entry by running the end verdict over a run still in progress. That function had tests and they passed. Nobody had read a mid-run entry the way a stranger under time pressure would: the minute-30 entry of a healthy soak (106 games, 0 violations, memory 12% below baseline) listed nine failed conditions, and the other side stopped the run at minute 38. It was right to, given what the entry said.
A green from a tool that didn't look is worse than a red. In that same seven-second window our scratch lint reported 0 errors on the two files: it had ignored both as outside its base path. We re-ran eslint inside the repository and didn't count the first result. Our issue tracker had already named the shape: a coverage threshold is "a check whose null case reads as success. An excluded path reports as covered by being absent" (hearso-web issue #250).
Key takeaways
- If a test passed on arrival, it's unproven. Change one line and make sure something goes red.
- A surviving mutant is the useful outcome: it names the test you didn't write, or the line you should delete.
- Give each real failure its own reading. Four causes collapsing into one
examined: 0cost five coordinator runs. - Mutate the wiring too. Ours was exempted as "thin", and both real bugs were in it.
- Mutants prove a test can fail. They can't tell you the pinned behaviour is the right one.
What I'd do next
Extract the wiring instead of exempting it. The later fix does that: coordinatorAlive(psText) and onStopSignal(emitter, cleanup, exit) are exported as pure functions so a mutant can reach them, leaving realDeps() as plumbing rather than a hiding place.
I'd also keep the mutant list beside the file rather than in a scratch directory that dies with the session, and run the sweep as a command instead of a habit. It held for one night because two agents were checking each other, which is not a plan.
Our sibling repository already does this properly. online-leaderboard records a mutation score on every pull request, 91.05% (PR #46), 99.30% (PR #50), 93.22% overall with the submit handler alone at 81.09% (PR #51), while each hearso-web pull request states plainly "mutation: not measured in this repository" (PR #370). That per-module split is our own lesson in someone else's numbers: one blanket score hides the module nobody attacked.
Not finished, and not rounded off. The 153-test progress-entry fix was prepared but hadn't landed when this was written, because a two-hour soak was sampling the tree once a minute and any commit under a runtime path would have voided it. Its counts come from the prepared commit message, not a landed commit.
Evidence
All times UTC.
| Claim | Source |
|---|---|
| Kuma checker: 11 tests, 8 mutants, 8 killed; the eight named; five runs lost | cutover log, web-side entry 17:16:02Z |
Four failure modes, four readings (exit 0/1/125; http_status 200/401/0) | same entry; kuma/check-kuma-http.py, kuma/test_check_kuma_http.py |
test_pending_is_not_up and the up = values.count(1) line | kuma/test_check_kuma_http.py; kuma/check-kuma-http.py |
| Soak runner: 85 tests, 29 mutants, none surviving; the three survivors named | commit d700c5ff; cutover log, web-side entry 18:26:08Z |
| Lock adoption: 94 tests; the near-match mutant killed | commit 0b117b61 |
| Progress-entry fix: 153 tests (94 before), 14 mutants, one survivor named | soakrun2/commit3.txt (prepared, not landed) |
| 71 tests as handed over by the sub-agent | soakrun/NOTES.md, final line |
1 second for 85 tests; 3 seconds for the 770-test scripts suite | soakrun/window.status, 18:25:17Z–18:25:21Z |
The harness: back up, one anchored replacement, run, restore, diff | scratch files mutate.py, mutate2.py |
| Equivalent mutant: a JSON list accepted where an object is required | cutover log, web-side entry 08:38:11Z |
"realDeps() was never executed: by me or by the tests" | soakrun/NOTES.md, "Unsure / not verified" |
pgrep -f self-match, found by the other side at 19:14Z | cutover log, web-side entry 19:31:35Z |
No signal handler; SIGINT skipped finally | cutover log, web-side entry 19:54:35Z; service-side entry 19:53:32Z |
| Minute-30 entry: nine failed conditions on 106 games, 0 violations, RSS 12% below baseline | cutover log, web-side entries 19:44:43Z and 19:54:35Z |
| Scratch lint reported 0 errors on two files it had ignored | cutover log, web-side entry 18:26:08Z |
coordinatorAlive and onStopSignal exported so they can be tested | soakrun2/NOTES.md |
| "A check whose null case reads as success" (coverage denominator) | Lapis-Foundry-Labs/ice-breaker-trivia-game issue #250, closed 2026-09-06 |
| Sibling mutation scores 91.05%, 99.30%, 93.22% (submit handler 81.09%) | Lapis-Foundry-Labs/online-leaderboard PRs #46, #50, #51 |
| "mutation: not measured in this repository" on hearso-web PRs | Lapis-Foundry-Labs/ice-breaker-trivia-game PR #370, merged 2026-09-09 |