Writing
Notes on building Hearso, the product decisions and the engineering ones, written as they happened. Every number in them comes from a run we can point at.
- The morning a July cron job stopped every container on my MacOne browser test out of seven came back 503. The code had not changed; the machine had, at 10:00:01Z, on a schedule I set two months earlier and forgot.
- req.nextUrl.origin vs the Host header: what a route handler may trustA browser check wanted 401 and got 403. The value our handlers compared against turned out to mean one thing on the platform it was written for, and something else everywhere else.
- One outside read, 70 files, one finding: the page inside two framesReviewing locally before pushing worked: one finding on 70 files. It was still a real one, because every check I had written asked that page what it said, never how it was framed.
- What we got wrong about checks that pass when nothing happenedA smoke alarm with a dead battery is silent in the same way as a house that isn't on fire. Eleven real instances from one night, four of them mine.
- How an answer reaches every screen in Hearso, and where its points goOne tap on an answer, two transports, and a leaderboard on AWS. Here is every hop, what is live, and what is still only running on a laptop.
- Break your own code on purpose before you trust its testsA test that passes the moment you write it is unproven. We spent a night breaking our own scripts one line at a time to find out which tests could actually fail.
- We threw away 38 healthy minutes of a two-hour soak testA two-hour soak of our new Rust WebSocket service passed on the fifth attempt. The four before it failed on sizing, on ordering, and once on a message our own tool wrote.
- 7 rules for two agents writing to one shared fileTwo agents spent a night building two halves of one system, talking only through a shared append-only log under a directory lock. Here's what broke, and what I'd set up differently.
- Why did our Uptime Kuma check stall before it could log in?A monitoring check kept failing for exactly ten seconds and telling us nothing. Getting to the bottom of it took three corrections and one prediction written down in advance.
- Line coverage said done. Mutation testing disagreed.The four modules we were most afraid of score 100%. The three handlers we actually ship score 56%, 70% and 76%. Here is what survived, and why coverage never told us.
- A benchmark is only as trustworthy as its namespaceA benchmark that cannot be replayed is not a baseline. Our first run left 100 probe players on every board it measured, so we rejected it and moved isolation into the config.
- Reads, not writes: what our rank histogram really boundsOur rank query stops counting at 2,000 and falls back to 128 log-spaced counters. It bounds what a rank read costs; it does nothing for writes, and I want to be clear about that.
- Anatomy of a score submission: every write it triggersOne score submission does not cause one write. We counted them, showed the cost arithmetic step by step, and then found the one number anyone has actually measured.
- We store every score backwards so DynamoDB never has to sortReading the top 25 of a leaderboard is one query with no sorting anywhere, because the sort key stores the score inverted. Here is the whole table design, the reasoning behind it, and the parts we still cannot do.