A check you have not seen fail for the right reason is not one yet
- date
- 20260816
- what happened
- In one afternoon of building the CI pipeline, two separate checks failed for the wrong reason and reported a confident, specific, false diagnosis: a background monitor announced a healthy host unreachable for ten minutes, and a CI step announced a lock file stale when it was not.
- what it cost
- Wasted debugging time chasing a healthy host and a healthy lock file. No production impact, but a check that lies with confidence is worse than one that says nothing.
- what changed
- The CI assertion was rewritten as a pure shell string comparison with no external command whose absence could be mistaken for a difference. The general practice adopted afterwards has two parts: cause the condition once and watch the check fire before trusting it, then break the check itself and confirm it fails differently from how the subject fails.
- the check now
- The CI practices runbook now says an assertion must not depend on its own reporting, the fleet is enumerated from the flake rather than hardcoded, and a discovery step that finds nothing must fail loudly.
Both failures are worth understanding mechanically, because both are easy to reproduce anywhere. The monitor watching sandbox1 ran under zsh, where an unquoted variable does not word-split the way it does under bash. It therefore tried to execute a literal ten-word filename as a single command, got command not found on every poll, and read that as the host being unreachable. The host was fine the whole time.
The CI step compared a flake lock file before and after a refresh using diff, which was not installed on the runner. The missing tool’s own error text, also command not found, satisfied the step’s negated test, so CI announced that the lock file was stale. It was not.
Two independent tools, a shell monitor and a CI gate, each failed at the level of the check itself being broken rather than the thing being checked being broken. Each produced language confident enough to send a reader straight to the wrong place. Learning this twice in one afternoon is what made it stick.
The fix in both cases was structural rather than a patch. Remove any external tool whose own absence can be misread as the answer, and adopt a standing discipline. Deliberately break a check and watch it fail in a way that does not look like the subject failing. A team running any unattended check, in CI or in a monitoring stack, can apply this without adopting anything else from this site.
Source: node0 lessons v0.1, lesson 6.4. Sanitized: checklist v0.1, 20260921; runbook file name; voice pass 20260921. Part of oznog.com/node0.