The day four failures went unnoticed, and none of them paged anyone
- date
- 20260906
- what happened
- On one day (20260906) two Ceph hosts had been hung for ten hours, four switches sat within a few degrees of their automatic thermal shutdown, a 12 TB storage disk had been dead for 38 hours, and a power event had left two machines dark. Every one of them was found by accident.
- what it cost
- Ten hours of degraded Ceph (131 of 170 storage daemons, 2 of 3 monitors); a real risk of four production switches shutting themselves down; 38 hours of undetected data-loss exposure on one disk; two hosts needing someone to press a power button.
- what changed
- The whole observability stack (Prometheus, Alertmanager, Grafana, Loki, VictoriaMetrics and eleven purpose-built exporters) was built in six phases starting that evening, each alert derived from a signal that had already existed and gone unread.
- the check now
- Firmware event-log polling on every Supermicro host, per-port flow-control frame rates on the access switches, switch thermal status, Ceph quorum and daemon counts, per-disk self-test and error counters with a passing self-test treated as no evidence, and power state tied to the inventory's power paths so an alert can name the hosts affected.
Four separate failures were in progress at once on 20260906, and the fleet said nothing about any of them. Two Ceph hosts had suffered a CPU internal error and hung for ten hours with chassis power on, network links up and their operating system journals simply blank. They were found because both machines started pushing roughly six hundred Ethernet flow-control frames a second into the switch fabric, which someone noticed while looking for something else. Four switches were running their physical-layer chips at 100 to 107 degrees Celsius against a 110 degree automatic shutdown, because a fixed fan-speed override set a few days earlier for noise had switched off the fan algorithm entirely. A storage disk had been dead for a day and a half, with nothing to ask about it except the drive’s own self-assessment, which read PASSED. And a power event had rebooted a rack, leaving two hosts without remote management dark.
The part worth copying is the response. Rather than designing monitoring from a blank page, Christoph and the agents went back through the day and wrote down, failure by failure, the signal that had already been on the wire and had gone unread: firmware event-log entries, flow-control frame counters, cluster daemon counts, power events from the UPS daemon. That list became the founding brief for the entire build, which started the same evening. If you are adding monitoring to a system that already exists, audit your last bad day for the data you already had, not the data you wish you had, and write the first alerts from that.
Source: node0 lessons v0.1, lesson 2.1. Sanitized: checklist v0.1, 20260921; serials, addresses, rack labels; voice pass 20260921. Part of oznog.com/node0.