A UPS service touch dropped a rack, and two hosts needed a person
- date
- 20260906, 17:53
- what happened
- During UPS servicing, one unit was unplugged, which dropped a rack's output. Both switches in that rack rebooted, seven hosts rebooted, and two of them, di3 and sandbox1, neither of which has a BMC, did not come back on their own and needed the physical power button.
- what it cost
- A physical visit to two hosts that would otherwise have stayed dark indefinitely, plus an unrelated discovery made only because the rack was being looked at. A third host had already been powered off for an unknown period and nobody had noticed.
- what changed
- The BIOS power-restore work that followed traces directly to this event. Monitoring gained a rule that a host missing from its expected inventory-derived set is itself an alert, not just a state change, and hardware events now get a journal entry on the device in NetBox as standing convention.
- the check now
- Host reboot detection, where a changed boot time since the last poll is itself an event, plus UPS and switch state-change monitoring. Both signals already existed on the day and simply were not being read.
This was planned maintenance, not a failure. A UPS was unplugged on purpose, for service, at 17:53 on 20260906. It still produced a five-minute outage across a rack, and for two machines it produced an outage that could not be ended remotely at all.
A BMC is a baseboard management controller, the small always-on computer on a server board that lets you power the machine on over the network when the machine itself is off. Two of the hosts in that rack do not have one. When their power came back they stayed in soft-off, and no amount of remote access could change that, because there was nothing powered to receive the command. Someone had to walk to the rack and press the button. A third host came back ten minutes later on its own, through a BIOS boot-order fallback rather than by design, which is not the same as working.
Everything on the other feed rode through untouched. Ceph, Unraid and both backup targets were unaffected. That is the design working. The gap was not redundancy, it was recoverability.
Two things came out of it. The first is an inventory question worth asking before you need the answer. Which of your hosts can be powered on remotely, and which of them turn any rack event into a drive? The second is a monitoring shape. The rack visit also turned up a host that had been off for an unknown length of time, unrelated to the event, which nobody had spotted, because nothing alerts on a machine that is simply absent. Monitoring now treats absence from the expected set as the alert, and every hardware event gets a permanent record attached to the device rather than living in someone’s notes.
Source: node0 lessons v0.1, lesson 5.6. Sanitized: checklist v0.1, 20260921; rack numbers; voice pass 20260921. Part of oznog.com/node0.