A knocked-loose USB cable went unnoticed for four days
- date
- 20260909 (lost), 20260913 (found and fixed)
- what happened
- One firewall's directly attached UPS dropped off its USB bus during unrelated work and stopped being monitored entirely. Nothing paged, because the failure mode was that the driver had nothing to talk to, not that the UPS reported a problem.
- what it cost
- Four days with no visibility into that firewall's power status, and its graceful-shutdown path unmonitored throughout. No outage happened in the window, and it was found by luck of scheduling rather than by any alert.
- what changed
- The cable was reseated, and the same physical check was extended on the same visit to the two other directly attached UPS units rather than treating the first as an isolated fluke.
- the check now
- None dedicated yet. This was caught by a scheduled physical inspection. The absent-client alert that would have caught it in minutes is the rule, not yet the check.
A small story with a sharp point. On 20260909, during unrelated work in the rack, the USB cable between fw1 and its own UPS came loose. The UPS kept working. The firewall kept working. The monitoring daemon simply had nothing on the other end of the bus, and said nothing about it. It was found on 20260913, four days later, because a scheduled three-hour maintenance pass happened to include a physical look at that exact cable.
The mechanism is the part worth internalising. Monitoring built around “alert when a device reports a problem” is blind to a device that stops reporting at all, because silence and health produce the same output: no alerts. A locally attached UPS whose data cable falls out does not raise a fault. It stops being asked anything. From a dashboard, that is indistinguishable from a quiet night.
The honest version of this lesson is that the fix was luck, and luck is not a check. What should have caught it is an absence check. A UPS that is in the fleet’s inventory and was previously reporting, and then stops, is itself the signal, independent of whatever state it last reported. That shape generalises well past power hardware, to any client, exporter or agent that reports in on a schedule. Alert on the gap in the stream, not only on the contents of the stream.
Node0 also did one thing right on the day of the fix that is worth copying. Having found one loose cable, the same physical check was run on the other two directly attached UPS units on the same visit, rather than treating the first as a fluke. A cause that knocked one cable loose during unrelated work had no reason to stop at one.
Source: node0 lessons v0.1, lesson 5.15. Sanitized: checklist v0.1, 20260921; hostnames; voice pass 20260921. Part of oznog.com/node0.