When Hearso breaks, the evidence is scattered across five tools. I wanted an AI assistant that could read all of them and change none of them.
Metrics and logs live in Grafana. The leaderboard's logs live in CloudWatch. Product analytics live in PostHog. Then there's the status page and the database.
During an incident, someone has to open all five, line the timestamps up, and decide what actually happened. It's slow, and it's exactly the kind of work an assistant should be good at.
It's also the kind of access that should make you nervous. Anything that can read your infrastructure is one bad credential away from changing it.
So the question was simple: can we give an assistant everything it needs to investigate, and nothing it could use to make things worse?
We built a small Model Context Protocol server to find out. I wrote the brief and its guardrails, a coding agent built it in a public TypeScript repository, and I reviewed every milestone and ran its first job.
Short answer: yes, but not on the first try. Review found two security bugs before it ran once, and one fix broke the checker meant to prove the fixes. Then, on its first real job, the most useful thing it did was refuse to call 288 zeros healthy.
Read-only has to be built in, not promised
The first decision came before any code: read-only by construction, not by promise. A read-only tool that relies on everyone remembering the rules is a write tool waiting for a bad day.
So the brief is thirteen groups of guardrails, G0 to G12. Each one needs a tagged test that goes red when the guardrail is removed. If a test can't fail, it doesn't count.
| Guardrail | What it stops |
|---|---|
| G0.1 read-only by construction | any HTTP method and endpoint pair not on an allowlist |
| G0.2 one exit | a tool result leaving without redaction, a size cap and an audit line |
| G0.5 output is data | an instruction planted in a log line being obeyed |
| G1 refuse writable credentials | a provider starting with a token that could write |
| G9 one gated write | a restart anywhere but a demo stack, without a two-step, single-use token |
Read as a whole, the table answers three questions in order: what may the server ask for, what could its credentials do if it asked for more, and what may leave. The one write, G9, is fenced off on its own. Figure 1 shows what that leaves the server able to touch.
Figure 1. The server's reach, from its guardrails G0 to G12 and the per-provider permission documents, 2026-09-28.
A role called "Viewer" can still write
G1 checks every credential at start-up and refuses to run a provider whose token can write. It met three very different situations.
Grafana's "Viewer" role is not read-only. Here's a real example. The first token I made used the built-in Viewer role, which sounds harmless. Its permission list carried 24 write-like actions: create, write, delete and execute on app plugins. The check refused it, which was the guardrail doing its job.
The fix was a service account with no basic role, plus four reader roles. Its token lists 9 actions, 0 of them writes. To prove the check could actually fail, I sent one real write, a folder create. It came back 403.
PostHog was the easy one. A personal API key can be limited to exactly query:read, insight:read, error_tracking:read and feature_flag:read, on a single project. Better still, the key can report its own scopes, so the server checks them at start-up and refuses anything wider. That provider is built and tested against synthetic data; it hasn't run against our live project yet.
The VPS host can't be made read-only at all. Its personal tokens carry the owner's full permissions, and there's no way to scope them down. A filtering proxy would still hold a full-power token somewhere, which is read-only by our code, not by the credential. So there's no provider for it. Host health reaches the server the other way round: the host pushes its own metrics to Grafana, and the server reads them with the reader token.
The lesson we took: a role's name is a label; its permission list is the fact. Ask what a token can do, then try one write that must fail.
Two security bugs, found before the first run
Each credential had now been asked to fail once. The code was next. The coding agent opened 14 stacked pull requests, one per milestone. Hosted CI was unavailable that day, so I ran every gate locally on each exact commit and posted the results as commit statuses. Then four review agents read each PR against its guardrails and tried to break each one by removing it. I checked every finding before it was posted (Figure 2).
Figure 2. The first review of the server's 14 pull requests, 2026-09-28 (UTC).
13 of the 14 came back with findings. The ones worth telling:
- Two security bugs, in our own guard around analytics queries. A query could escape its time window with unbalanced parentheses:
1=1) OR (1=1. And redaction trusted column names the caller chose, soSELECT person.id AS eventcame back unmasked. The guard now accepts one narrow shape: a singleSELECTfromevents, fixed projections, no aliases, balanced parentheses, and a time window the server inserts itself. Returned cells are classified by that validated shape, never by the labels that come back. - Usage counts inflated up to 3 times, because deduplication keyed on timestamps rather than message ids.
- Several tagged guardrail tests could not fail. Remove the guardrail they named, and the whole suite stayed green.
The leak scans came back clean: 0 secrets in 37 commits, and 0 hits for a 15-term private denylist. That second zero only means something because we planted a term on purpose and the scan found it. (We'll come back to zeros.)
The pattern I see most is the last finding. A test that has never been seen to fail has not been shown to work.
The fix that broke its own checker
The remediation PR fixed the findings: 3 commits, 44 files, 109 tests, 92.34% line coverage. My independent run reproduced those numbers, and the query guard refused all 14 bypass variants the re-review threw at it.
It also added a new bug. We learned about it the annoying way.
The guardrail-traceability checker had been rewritten, and it now silently skipped any table row it couldn't parse. A guardrail with no test at all, marked pending, produced this:
Guardrail traceability passed: 55 active IDs.
The root cause was two regular expressions over the same table: one chose the rows to check, the other counted the rows to report. When they disagreed, the difference just vanished. The fix is to fail on any row that doesn't parse, and to report only what was actually checked.
It is the same shape as the tests that could not fail, one level up. This time the thing that checks the checks passed on a row it had not read.
Seven earlier findings were still open after the re-review, so nothing has merged yet. That's fine. It's what review is for.
My first guess was wrong
The first real job was small on purpose: verify one 3-player, 10-question WebSocket test game on DEV, using only the read-only token. With nothing merged, the server that did it was a local checkout of the remediation branch at commit 8079b27, with only its Grafana provider switched on.
Before handing it over, I queried the logs myself. I found 2 lines for the room and 0 carrying a player reference, and concluded the temporary game container's logs were never shipped.
They were. I had filtered on a label that only compose-managed containers carry. With the right selector, the room had 95 lines (Figure 3). My zero came from my query, not from the logs. It was the denylist scan without its planted term: a zero with nothing beside it to show the query could have found anything.
Figure 3. The two zeros in the first run, one 3-player test game on DEV, 2026-09-28 (UTC).
The most useful answer was "I can't tell"
Here's what the server reported, next to my independent check:
| It reported | Independent check |
|---|---|
| 95 lines: 93 from the game service, 2 from the relay | 95 |
| 67 seat-level lines, all carrying a player reference | 67 + 4 from a second test room + 1 start-up line = my 72 |
| 0 raw player ids visible | 0 |
| 288 metric samples, all zero: "cannot establish zero traffic" | the test container was never scraped |
The 67 against my 72 looked like a disagreement at first. It reconciled exactly, and the reconciliation is what made the rest believable.
But the last row is the reason we built this. The metric query returned 288 zero-valued samples. A tool eager to please would report "0 errors, 0 requests, healthy." The server wrote that these zeros measure nothing for this game, because the container they'd describe was never scraped.
It also listed everything it couldn't examine, and it included its own two failed first attempts: one where it used display names instead of data-source ids, and one where it ran my bad selector. Both sit in the report next to the corrected runs.
The server exists to say what happened during an incident. A zero it cannot vouch for is not part of what happened, so the right place for it is the list of things not examined.
An ops tool is judged less by what it can count than by the conclusion it refuses to draw.
Predictions vs outcome
| Prediction | What happened | Right? |
|---|---|---|
| The Viewer role would be enough for read-only | 24 write-like actions; needed a custom service account | No |
| The host API could be wrapped read-only | its tokens can't be scoped; no provider built | No |
| Review would find style issues, not security bugs | 2 security bugs in the query guard | No |
| The fixes would be clean | one fix broke the traceability checker | No |
| The test container's logs were missing | they were there; my selector was wrong | No |
| The server would refuse to read unscraped metrics as health | it did, and said why | Yes |
Five out of six wrong, and every one of them was caught before it mattered. I'll take that trade.
What it cost
From the review agents' own completion records. The coding agent that wrote the server isn't included; I haven't measured its usage yet.
| Work | Runs | Tokens | Tool calls |
|---|---|---|---|
| First review | 4 | 813,104 | 145 |
| Re-review of the fixes | 3 | 449,398 | 100 |
| The verification game | 1 | 419,102 | 26 |
| Total | 8 | 1,681,604 | 271 |
Key takeaways
If you're building a tool an AI assistant will use against real infrastructure, steal these:
- Check the permission list, then prove a write fails. A role's name is a label.
- If a credential can't be read-only, don't hold it. Find another way to get the data to you.
- Make every guardrail test fail once, on purpose. A test that can't fail is noise with a green tick.
- Report what was checked, not what was counted. Our checker counted rows it never read.
- Give the tool permission to say "I can't tell." Then give every zero a positive control, because empty and broken look identical.
What I'd do next
- Close the remaining findings and merge the remediation PR.
- Run the server's own usage report (G12, counts only, never message text) on its build, and publish the coding agent's numbers beside the ones above.
- Connect the PostHog provider to our live project once our analytics setup is back in shape, and point the server at production WebSocket rooms when they go live.
Evidence
All times UTC, 2026-09-27 to 2026-09-28.
- Guardrails and reach: the server's brief (G0 to G12) and its per-provider permission documents, including the PostHog scopes and the start-up scope check.
- Credentials: the handoff log. The Viewer token had 24 write-like actions; the reader service account had 9 actions, 0 write-like; the folder create returned 403.
- First review: the 14 posted review comments; the secret scan over all refs (37 commits, 0 findings); the denylist scan (15 terms, 0 hits, positive control found).
- Remediation and re-review: the remediation PR (3 fix commits, 44 files; 109 tests; 92.34% lines); the re-review comments (14 bypass variants refused; the traceability checker's two regexes, read in its source).
- First job: the server's verification report and its two redacted evidence files (the report names the local stack's head,
8079b27, and says the server ran locally with only its Grafana provider); my Loki counts with the corrected selector (95 lines, 93 + 2; 72 = 67 + 4 + 1). - Cost: the review agents' completion records.
No key value, account id, project reference, email address, player identity, room code or player reference appears in this post or its figures.