Monitoring and alerting

This theme starts on 20260906, the day Node0 found two hosts hung for ten hours, four switches minutes from an automatic thermal shutdown, a disk dead for a day and a half, and a rack that had rebooted itself, all at once and all by accident. Nothing paged anyone. The whole monitoring stack was built in the weeks after that, and these lessons are what was learned while building it, mostly by getting it wrong first.

They fall into three groups, and they arrived roughly in that order. The first group is about signals that exist but are not being read: the hidden device-statistics log on the fleet’s most common drive model, the firmware event log on every server, the flow-control counters on the switch ports. Nearly every failure of 20260906 had already announced itself somewhere; the founding brief was written by listing those announcements rather than designing from a blank page.

The second group is about monitors that are themselves broken, which turned out to be the larger problem. Four monitors written in one week all had defects, and the silent ones were the dangerous ones. A monitor that polls over a login session faulted the very thing it was watching. An alert keyed to a device path paged seven times for drives that had not changed. An alert built on a wrong mental model reported a healthy standby as a fault, continuously. In every case, reading the check for correctness was not enough; only causing the condition proved anything.

The third group is about the alerting path as a system in its own right, with its own failure modes: a folder permission that silenced everything for 98 minutes, a quorum threshold that a fourth cluster member made silently wrong, rules shipped before the thing they watch existed, a maintenance silence that muted almost nothing, and the discovery that Node0’s monitors, inside its network, could not tell a closed port from an exposed one at its own edge, so exposure is now checked from outside.

The three later additions, from July and August (2.19 to 2.21), predate the build and were folded in because the rest of the theme kept arriving at their conclusions independently.

The monitoring as it stands now is on the observability layer page.

Declined #

• The habit of reading fire-and-resolve notification pairs together as a single incident (20260912): a change in how someone reads a channel, not a change to a system, tool or runbook. Folded into the dead-man’s-switch lesson as context.
• A dashboard-embedding quirk in the storage cluster’s Grafana folder (20260911): a one-time documentation note about an exception to a convention, with no incident and no cost attached.
• Two permanently unreachable scrape targets removed from monitoring (20260910): a clean, uneventful correction with no cost and no near-miss.
• Windows event logs reaching the log aggregation system: listed as open at the capture of 20260920, and flowing when checked on 20260929, with events from the Windows server in the preceding hour. Declined because it never became a dated event with a cost and a fix.
• The Ceph maintenance-mode procedure lapsing a second time, on a different host, after having been fixed once: an adherence lapse rather than a defect in the monitoring system. The check that catches it is procedural discipline, not a new alert.