Skip to content
Oznog

2.18 · monitoring and alerting · as-is

A monitoring runbook's own numbers were stale prose, not live truth

date
20260920
what happened
An audit of the monitoring layer's documentation against the running system on 20260920 found several small mismatches: the operator runbook still said seven dashboards where there were now fifteen, a draft state capture had quoted rule, test and publisher counts from memory and every one was off by a handful, and the dead-man's-switch design said about ten minutes where the last measured value, recorded elsewhere, was closer to thirteen.
what it cost
Nothing acute, but a slow accumulation of individually harmless inaccuracies that compound into a runbook nobody can fully trust without re-verifying it.
what changed
The affected sections were rewritten either to state numbers as of a specific dated measurement, or to be recomputed from the live system at the time of the audit, with a standing preference for counts that are queried rather than typed.
the check now
Any operational number quoted in a runbook without a date attached is treated as a claim to be re-verified, not a fact to be trusted. Where practical, a hand-typed count is replaced by a command that recomputes it.

None of these mismatches were dangerous on their own. A runbook saying seven dashboards when there are fifteen costs nobody anything directly; it is simply a sentence written when the count was seven, which nobody had reason to revisit as integrations were added one at a time. The design document describing a dead-man’s-switch response time as about ten minutes was similarly innocent. Ten minutes was the target, and the measured value of just over thirteen lived in a different document written by a different pass.

The pattern is what makes it worth recording. Every number in operational documentation is one of two kinds. Some describe a fixed design decision, and those stay true until the decision changes. The rest describe the current state of a living system, and those have a shelf life, which is invisible to the reader unless the number carries a date. A reader has no way to tell from the page whether seven dashboards is this morning’s count or last spring’s.

Two practices came out of the audit. Every operational number now carries the date it was measured, so the reader can decide for themselves whether to trust it. And where a count can be recomputed, the runbook prefers a command the reader can run over a figure someone typed once, because a command cannot go stale.

The transferable form is simple enough to adopt today. Treat any undated operational number in your own documentation as a claim rather than a fact, and prefer numbers a reader can recompute over numbers someone remembered.

Source: node0 lessons v0.1, lesson 2.18. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.