What we got wrong about checks that pass when nothing happened

I wanted to know how many of our release checks would look the same whether or not they had actually run. This is the count, the list, and what replaced them.

Why I went looking

Hearso is a small real-time multiplayer trivia platform. On 2026-09-18 a web-side agent and a service-side agent spent a night moving live rooms from HTTP polling onto a new Rust WebSocket game service, gated by a loop of nine automated checks and a two-hour soak.

By morning we'd written eleven separate findings of one shape into the shared log. Four were in my own code. None of what follows needs any knowledge of the project.

The shape

A smoke alarm with a dead battery is silent in exactly the same way as a house that isn't on fire. You can't tell them apart from the alarm, and silence is what you were hoping for.

Precisely: a check has a null case when what it reports on finding nothing to look at is indistinguishable from what it reports when everything is well. Our log check is the plain example. It queried Loki, got nothing back, and wrote {"examined": 1, "metrics": {"streams": 0, "errors": 0}}, because its count was max(1, len(streams)): "one query was made".

Zero ERROR lines is true of a log store with no lines in it. And because examined could never be 0, the coordinator's own rule that examined 0 means "not configured" could never fire. That leg had been passing over an empty database all night, and it was one of the nine we planned to bet a release on.

Figure 1. The same pipeline twice, both rows real legs of the 2026-09-18 release loop. The difference is where the count comes from.

Eleven of them in one night

The check	What it printed	What it had actually examined
Monitor check	examined 0	one of three causes (missing variable, unreachable server, login never delivered) all with the same reading
Config loader	nothing at all	a file that wasn't there; except OSError: pass
Invariants check	{"examined": 2, "metrics": {"sim_seeds": 10000, "tests": 2}}, byte-identical in six run folders	a 43-line arithmetic loop with no room engine, no database, no service
Log check	{"examined": 1, "streams": 0, "errors": 0}	an empty log store
Ops console redaction test	pass	two string constants that can't contain what the test forbids
soak.sh	"complete"	nothing: every command in its loop ends || true
invariants.sql	never ran	tables and columns that don't exist in this schema
My before/after comparison	"6378 passed, 0 failed" on both sides	two identical environments; the change hadn't arrived
My scratch lint	0 errors	0 files: both were outside its base path
My liveness check	"the coordinator is alive"	my own monitoring shells, which mention the coordinator's path
My progress report	nine conditions "Failed"	numbers the workload only hands over when it exits

Figure 2. Every instance verified in the cutover log on 2026-09-18 (UTC), n = 11. Four were ours, including both of the two that cost measurable time.

The last two are the mirror image, and they deserve the same attention. My liveness check asked pgrep -f <path to the coordinator>, which matches any process whose command line contains that text: including my own monitoring shells. It answered "alive" when nothing was, and a launcher waited for ever.

None of this was new, which is the part I find hardest to write down. The same shape had already shipped four times in the web repo in the previous three weeks:

• An accessibility scan treated axe's incomplete state as neither pass nor fail, so twenty unmeasured nodes looked like a clean page (PR #106).
• A protected browser test had a gate that fired on every event and never once ran: "all 48 runs are skipped" (PR #107).
• A review gate's success status covered completed, skipped and rate-limited reviews alike. 19 of 24 pull requests merged on it (PR #269).
• A scheduled sweep ran every five minutes for a day and collected 0 of 12 qualifying stale rooms, because it matched the wrong column (PR #425).

Figure 3. The eleven instances of 2026-09-18 (UTC) in the order they were written down. They arrived at roughly one every forty minutes across seven hours, not in one batch.

What they cost

The worst was mine. My soak runner's progress entries ran the end verdict over a run still in progress. The workload only hands over its totals when it exits, so mid-run every workload number is -1, "not measured", and -1 correctly fails a condition that has to be zero at the end.

So a healthy soak was stopped at minute 38. At that moment it had finished 106 games, with 184 reconnects, 43 player leaves, 28 settings changes, zero invariant violations, zero 1011 closes, zero unexpected closes, zero HTTP 5xx, a snapshot p95 of 1 ms, and memory 11.69% below its minute-15 baseline. The entry beside it listed nine conditions as "Failed". The service-side agent read it, stopped the soak, then diagnosed it exactly right. A line that says FAIL will be acted on.

The || true soak cost differently. With no exit code read anywhere, its bot loops spun at full speed against a server that had stopped answering: 3,495 evidence files in the first attempt (10 games played, 3,485 failed at setup) and 2,856 in the second, every one a failure. Each failed attempt still created a room and seated four players before giving up, so it wrote 3,200 rooms and 11,787 player rows into the local database in about 100 seconds. Two hours of that is roughly 370,000 rooms in a database every check leg reads.

The invariants check is the one that would have shipped a false claim. Its evidence says 10,000 seeds, which reads like a chaos harness; the 10,000 is the argument to a loop that adds non-negative numbers to a counter and asserts the counter isn't negative.

# the test's exit code is taken, then this is written whatever happened
evidence.write_text(json.dumps({"examined": 2, "metrics": {"sim_seeds": 10000, "tests": 2}}) + "\n")
return result.returncode

If that package ever ran zero tests and exited 0, the leg would still report examined 2, sim_seeds 10000.

The rules I use now

A check has to state what it examined, not only its verdict. That one sentence covers all eleven. What follows is what we did that night, not a wish list.

Count from the thing, and let the count move. The replacement invariants leg reads the database, and its examined went 42, 46, 50, 54, 58, 62 and on to 101 across consecutive runs, because rooms kept being played. A number that moves is a number that was taken; the old leg's 2 never moved.

Zero and "not measured" must never pass. In the soak runner a zero-condition is one line, and "not measured" is -1 so that it fails that same condition:

const NOT_MEASURED = -1;
/** A condition that passes only at zero, so -1 ("not measured") fails it. */
const zero = (name, value) => ({ name, value, pass: value === 0 });

Don't let an empty set answer for a healthy one. The maximum of an empty list of database samples is zero, and zero violations is what a passing soak looks like. So the number of samples that produced no reading is its own condition, and so is the share of minutes that were sampled at all.

Give each cause its own number. The replacement monitor check returns four readings for four states, verified against the real server: real credentials exit 0, examined 7, http_status 200; wrong password exit 1, http_status 401; no credentials exit 125, configured 0; a URL nobody serves exit 1, http_status 0.

Keep a control that must not succeed. The no-credentials 401, answered in 0.02 s, proves the request reached something that cares about credentials. When counting a process's settings I read two that every process has first, which have to be 1, before believing a 0.

Prove the change arrived before deciding it had no effect. My worst instance was a before/after where the "after" never happened, reported with real numbers. Now I count variable names in the child's own environment first, with a control name that has to read 1.

Break the code on purpose and check a test notices. That's mutation testing: you change one thing in the code under test (flip a comparison, delete a clause) and any test that still passes wasn't watching that behaviour. Our replacement monitor check has 11 tests, and 8 mutants killed 8 of them in turn; two of those mutants are precisely the Loki defect, "examined floored at 1" and "an empty server reads green". The soak runner took 29 mutants with 3 survivors, and each survivor got the test it was missing before the file was committed.

We do this by hand, per fix, and I'd rather it were a gate. Our leaderboard service has scored mutation since its first benchmark work (91.05%, then 93.33% with a per-module breakdown) while the web repo's CI badges say plainly that nothing measures mutation there yet (PR #193). The same day as this cutover, a keyboard-control fix in the web repo shipped with a five-mutation table showing 12 tests killed across the five (PR #429). Ad hoc proof beats none, but a number that only appears when someone remembers to produce it is not a gate.

Key takeaways

• When a check reports success, ask what its output would look like if it hadn't run at all. If the answer is "the same", it isn't a check yet.
• Evidence should carry a count taken from the thing examined. A constant is a decoration.
• Never let 0 or "not measured" pass a condition; give "not measured" its own value, and make it fail.
• Give every failure cause a distinct number, and keep one control that must not succeed.
• Read every generated message as a stranger under time pressure would. Mine said FAIL, and a healthy run was stopped.
• Expect these in your own work. Four of the eleven were mine, and both of the costly ones were found by somebody else.

What we would do next

Three changes are queued, all tooling only, all test-first: progress entries show live counters and mark end-only conditions "pending" rather than FAIL; the runner releases its lock and stops its server on SIGINT and SIGTERM, which it didn't do; and the liveness check stops matching shells that merely mention a path.

Beyond that, one cheap habit. For each check in a diff, write one sentence saying what its evidence would say if the system under it were missing, empty or unreachable. Every one of the eleven above would have been caught by that sentence.

To be fair to the service-side agent: seven of the eleven were in its tooling, and it fixed each within minutes of the finding being posted (a separate exit code for a missing credential, the loader path, monitors pointed at reachable addresses) and it aborted two of its own soak runs itself. It also found both of mine that cost time. This isn't a property of one author. It's a property of checks written quickly against systems that aren't there yet.

Evidence

All times UTC, all 2026-09-18. Paths are relative to the cutover working folder or the web worktree.

• The eleven instances and the entry that recorded each: cutover log, web-side entries 12:52:33Z (constant invariants evidence), 14:14:00Z (Loki examined floored at 1), 15:58:52Z (a redaction test over two constants), 16:46:22Z (three causes, one reading), 16:58:13Z (the config loader and my own empty before/after), 17:52:11Z (soak.sh and invariants.sql), 18:26:08Z (my lint over zero files), 19:31:35Z (my pgrep -f liveness), 19:54:35Z (my progress entry).
• Constant invariants evidence: bin/check-invariants.py; six folders under var/green-runs/*/invariants.json hold byte-identical {"examined": 2, "metrics": {"sim_seeds": 10000, "tests": 2}}.
• The 43-line loop: crates/sim/src/lib.rs; its two tests assert run(200) == 200 and run(10_000) == 10_000.
• The Loki leg: bin/check-loki.py (examined = max(1, len(streams))); var/green-runs/20260918T135900536822Z/loki.json = {"examined": 1, "metrics": {"streams": 0, "errors": 0}}.
• The monitor leg's single reading: bin/check-kuma.py, the except Exception branch writing {"examined": 0, "metrics": {}}.
• soak.sh: bin/soak.sh, 48 lines; || true on the bot loop (line 26), k6 (33), invariants (38) and the monitor check (39), under set -euo pipefail; the end entry asks a human to "inspect every bot".
• invariants.sql: names public.outbox_events, room_id, delivered_at, rooms.phase; the real names are public.game_finished_outbox, room_code, processed_at, rooms.status. No script or runbook references the file.
• Soak artifact counts: var/soak/20260918T184007Z holds 3,495 bot evidence files, 3,485 of them setup_failed; var/soak/20260918T184550Z holds 2,856, all setup_failed. The 3,200 rooms and 11,787 player rows counted in the local database between 18:45:50Z and 18:47:30Z are in the cutover log, web-side entry 18:48:21Z.
• The stopped soak's health at minute 38: cutover log, service-side entry 19:53:32Z and web-side entry 19:54:35Z; the nine "Failed" lines and rss_growth_percent = -11.69 are in the progress entry of 19:44:43Z.
• The moving examined: var/green-runs/*/invariants.json after the database-backed leg landed. 42, 46, 50, 54, 58, 62 … 101 Across consecutive runs; the same numbers appear in each run's coordinator entry as invariants_examined.
• NOT_MEASURED and the zero helper: scripts/cutover/soak-run.mjs, tested by soak-run.test.mjs. Landing it raised the web check's test count from 6378 to 6463, visible in var/green-runs/20260918T182523740851Z/web.json.
• The four modes, the 401 control and the mutants: scratchpad/kuma/check-kuma-http.py and test_check_kuma_http.py (11 tests); the four real readings and the 8-of-8 mutant result are in the cutover log, web-side entry 17:16:02Z. The soak runner's 29 mutants and 3 survivors are in the web-side entry 18:26:08Z.
• The liveness fix proven in four states: cutover log, web-side entry 19:31:35Z.
• The four earlier instances in the web repo: PR #106 (2026-08-26, twenty axe incomplete nodes "neither passing nor failing"); PR #107 (2026-08-26, "all 48 runs are skipped"); PR #269 (2026-09-06, nineteen of twenty-four merged on a success status covering completed, skipped and rate-limited); PR #425 (2026-09-18, 12 qualifying stale rooms measured, 0 swept, job running every 5 minutes).
• Mutation as a gate versus ad hoc: leaderboard-service PRs #46-#48 (2026-08-18, 91.05%) and #52 (2026-08-19, 93.33% with a per-module breakdown); hearso-web PR #193 (2026-09-04, badges state nothing measures mutation yet); hearso-web PR #429 (2026-09-18, five mutations, 12 tests killed across them).

Nothing above contains a credential, a token or personal data. Every environment inspection cited counted variable names only.