A UPS running its own self-test looks exactly like a real power event, and nothing was watching UPS state at all
- date
- 20260907 evening, found 20260908 04:00
- what happened
- On the evening of 20260907 a UPS ran its normal periodic self-test and briefly put its load on battery. The monitoring plugin on the primary storage host raised three back-to-back warnings that the fleet's main storage server would shut down in 240 seconds. It was found four hours later, by accident.
- what it cost
- A credible false warning that the storage everything depends on was about to shut down, with no real event behind it, and an unknown-length window in which a genuine power loss on that host would also have gone unnoticed.
- what changed
- UPS status and battery charge became a monitored signal for all four units, reporting only on state change, with the rule that a battery-discharge state means nothing unless it is paired with a real transfer reason.
- the check now
- An on-battery state is never alone sufficient to alert; a scheduled self-test is suppressed by matching its specific reason code, never by ignoring on-battery generally. Low battery always pages. Loss of communication with a UPS is worded and routed as an unknown state, so it cannot be mistaken for a real outage.
A UPS doing exactly what it is supposed to do, running a scheduled self-test every few weeks, raised the loudest alarm available to it: imminent shutdown of the storage host that everything else in the fleet depends on. Nothing was wrong. The monitoring plugin saw a battery-discharge state, concluded power had been lost, and started the shutdown countdown message.
False alarms are corrosive in one specific way, and this one is a textbook case. It teaches the reader to dismiss the next one, which may be real. The fix is not to stop watching for battery state. It is to stop treating battery state as the alarm. A UPS reports why it transferred to battery, and a self-test reason is a different value from a real utility loss. The rule now is that an on-battery state alone means nothing; it must be paired with a transfer reason that indicates a genuine event.
The second half of the story is the more useful half. Finding this also revealed that in a week when switches, disks and the compute cluster had all gained monitoring, electrical power itself had none. Nobody had decided not to watch it; it had simply never come up. The habit worth taking from this is to enumerate what your monitoring deliberately does not cover, as a written list, rather than discovering the gaps one accident at a time. Power, cooling, and the reachability of the management network are the classic blind spots, precisely because they are infrastructure for the monitoring system rather than things the monitoring system watches.
Source: node0 lessons v0.1, lesson 2.4. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.