Line coverage said done. Mutation testing disagreed.

I wanted to know which parts of this service the tests actually check, rather than which parts they merely run.

Why coverage was not enough

Our leaderboard service is small; the test suite is not. 611 tests, no failures, 99.54 percent statement coverage in the last CI record I have. That comfortable number means one thing: nearly every line got executed while the tests ran. It says nothing about whether anything would have complained had that line been wrong. So I ran mutation testing, using vitest with its v8 coverage and Stryker driving the same suite.

What mutation testing is

Imagine proofreading by deliberately introducing typos and seeing whether the proofreader catches them. If nobody notices, you have learned something about the proofreader, not the typo.

Mutation testing makes one small mechanical change to your source (a > becomes >=, an object literal becomes {}, a string becomes empty) and reruns the whole suite. If a test fails, the mutant is killed and that line is genuinely checked. If every test still passes, the mutant survived: nothing you wrote would have caught the change. The mutation score is the share of mutants killed.

Our run generated 2,349 mutants across 26 files and killed 90.42 percent of them, which clears the configured break threshold of 90. Then you look at it file by file.

Figure 1. The mutation loop, using one survivor from our own run: the structured log payload of the malformed-body error was replaced with an empty object and no test noticed.

The split

File	Statement coverage	Mutation score
handlers/query-leaderboard-handler.js	97.56%	56.58%
handlers/seed-simulator-handler.js	100.00%	69.90%
handlers/submit-score-handler.js	98.10%	76.45%
lib/score-key-builder.js	100.00%	100.00%
lib/leaderboard-id-parser.js	100.00%	100.00%
lib/utc-period-calculator.js	100.00%	100.00%
lib/score-rule-validator.js	100.00%	100.00%

Those last four are the one-way-door modules: the key builder, the ID parser, the period calculator and the score validator. Get one wrong and the mistake is written into every stored row, so they carry a 90 percent mutation gate and property-based tests. All four sit at 100.

The handlers are where the service meets the world, and they are the weakest thing in the repository. Coverage never hinted at it: seed-simulator-handler.js runs every statement it has and still loses 30 percent of its mutants.

Figure 2. Mutation score for 13 of the 26 files in the run committed 2026-08-19, recomputed from the report's per-mutant statuses. Our written gates are 90 percent on the one-way-door modules and 80 percent elsewhere.

What survived, and what it means

The handler that accepts every score generated 259 scored mutants. 56 survived and 5 were never executed at all. Grouped by the kind of change:

Kind of change	Survived
Object literal emptied ({ … } → {})	14
String literal emptied	12
Optional chaining removed (body?.score → body.score)	9
Conditional forced to true or false	8
Logical operator swapped	6
Arithmetic operator swapped	6
Boolean literal flipped	1

Read together, those tell one story. Almost every emptied object and string is a structured log payload or an error message, because our tests assert the HTTP response and never the log line. Every removed ?. is on the request body, because every test sends a well-formed body, so the defensive chaining never carries weight in a test.

Figure 3. Mutant outcomes for five files in the same run, counted from the report's per-mutant statuses. "No coverage" means no test executed that line at all.

Most of that is cosmetic, and I won't pretend otherwise. Two survivors are not.

The retry backoff, 25 * 2 ** (attempt - 1) + Math.floor(random() * 25), survives having its + turned into a - and its random() * 25 turned into random() / 25. Our tests inject a fake sleep and never assert what was passed to it, so any backoff policy at all would pass.

The second bothers me more. The handler computes Math.floor(row.submittedAtMillis / 1000) for epoch seconds, and that value feeds a condition asking whether an admin's write lease has expired. Turning the / into a * survives. A number a million times too large makes every lease look expired, and that check is what stops a score write landing in the middle of an admin operation. No test tells the two apart.

The gate I got wrong

Our written rule is 90 percent mutation score on the one-way-door modules and 80 percent everywhere else. Stryker's threshold is a single number for the whole run, and the whole run scored 90.42 percent. So it passed, while three handlers sat at 56.58, 69.90 and 76.45 percent, all under the 80 we wrote down for ourselves.

The rule exists in prose and nothing reads it. Our config carries one threshold for the whole run, so the number that would have caught this (76.45 on the busiest handler) never becomes a build failure. That is a gap in our process, not in the tool.

The aggregate can also move for reasons that have nothing to do with test quality. An earlier run the same day scored 93.33 percent over 22 files; this one scored 90.42 over 26. Nothing got worse: four more files entered the mutated set, two of them handlers.

Key takeaways

• Coverage counts lines executed. Mutation testing counts lines checked. Ours differ by more than twenty points on the handler that takes every score.
• An aggregate gate hides its worst file. 90.42 percent passed a 90 percent threshold with three handlers below our own floor.
• Surviving mutants cluster where assertions stop: log payloads, error strings and defensive ?..
• Read the survivors, not just the score. Two of our 56 are real holes; the score alone cannot tell them from the other 54.
• The modules we were most frightened of are our best-tested ones. Fear is a decent guide to test effort, but it stopped at the module boundary.

What we would do next

1. Make the 90/80 rule machine-checkable: score each file from the report's per-mutant statuses and fail the build on the worst one, not the average.
2. Assert the backoff. Capture what the injected sleep is called with, so a policy change has to be deliberate.
3. Write one test proving the lease condition refuses a write during an active lease, with a wrong-unit timestamp as the negative case.
4. Decide, in writing, whether log payloads are contract or convenience. Our operator docs treat them as contract, so some of them need assertions and the rest can be excluded on purpose.
5. Chase the 47 mutants with no coverage before the ones that merely survived. Those are lines no test runs at all.

Evidence

All times UTC.

• 2,349 mutants across 26 files, whole-run score 90.42%: reports/LFL26LEADBORD-055/mutation.json, committed 2026-08-19 in 99dc66c ("refresh mutation evidence"). The file stores a status per mutant, not a percentage; every score here is recomputed with Stryker's own formula, (killed + timeout) ÷ (killed + timeout + survived + no coverage), which excludes ignored mutants.
• Per-file scores: same file. query-leaderboard-handler.js 43 killed / 8 survived / 25 no coverage = 56.58%; seed-simulator-handler.js 72 / 16 / 15 = 69.90%; submit-score-handler.js 197 killed + 1 timeout / 56 survived / 5 no coverage = 76.45% (16 ignored mutants excluded); rank-percentile-calculator.js 141 + 1 / 1 / 0 = 99.30%; the four one-way-door modules 91, 107, 124 and 150 mutants, all killed.
• 611 tests, 0 failures, and statement coverage: reports/ci/vitest.json, run started 2026-08-18T19:42:00.970Z. The 99.54% total and the per-file statement figures are computed from that file's coverageMap; the file records hit counts, not percentages.
• Thresholds: stryker.config.json: {"high": 95, "low": 90, "break": 90}, applied to the run as a whole. The 90/80 per-module rule lives only in the repository's written engineering standards.
• Survivor categories and the two that matter: reports/LFL26LEADBORD-055/mutation.json, submit-score-handler.js mutants with status Survived; the backoff expression and Math.floor(row.submittedAtMillis / 1000) both appear there with their mutated replacements. The lease condition they feed is attribute_not_exists(#ownerToken) OR #leaseUntil < :nowEpochSeconds in src/lib/leaderboard-metadata-counter.js.
• 47 mutants with no coverage: summed across all 26 files in the same report (2,092 killed, 4 timeout, 175 survived, 47 no coverage, 31 ignored).
• The earlier 93.33% over 22 files: reports/LFL26LEADBORD-035/mutation.json, recomputed the same way; reviewed as Lapis-Foundry-Labs/online-leaderboard#52, which quotes the same figure with its per-module breakdown.
• rank-percentile-calculator.js is absent from that coverage record: it was committed later the same evening, so no statement-coverage figure for it is quoted above.