A monitor that logs into the thing it watches becomes part of the fault it reports
- date
- 20260909
- what happened
- On 20260909 the UPS management host began reporting rolling loss of communication with all four UPS units, one after another, with no power event anywhere. A monitoring probe was polling UPS state over SSH, and each login spawned and tore down a full user session on that host.
- what it cost
- An unnecessary rolling false alarm across the whole UPS fleet, and a real risk, because the host's shutdown-orchestration logic treated no communication the same as power actually lost, so a monitoring artefact could have coincided with and worsened a genuine event.
- what changed
- The probe was rewritten to speak the UPS management protocol directly over one lightweight network round trip instead of opening a login session per poll, and it now requires three consecutive failures, worded as an unknown state, before it says anything.
- the check now
- Measure a monitor's own footprint on the thing it polls before trusting its output, especially anything that opens a new session, a management-controller session, or a connection to a device that tolerates only one session at a time.
Two incidents that week had found monitors that lied about the system. This one found a monitor that changed it. The UPS management host started reporting that it had lost communication with each of the four UPS units in turn, which is precisely the shape of a real, spreading electrical problem. There was no electrical problem. A probe was checking UPS state by opening an SSH session to the management host roughly every 15 seconds, and each of those logins is not a cheap operation: it creates a full user session, runs the login machinery, and tears it back down. That churn was enough to starve the UPS driver processes running on the same host until they timed out, one at a time.
The specific mechanism is almost incidental. The transferable question is: what work does this polling mechanism do to its target simply by existing? A device that accepts only one management session at a time, a service tuned for an occasional human at a terminal rather than constant automated polling, a management controller with a small processor. All of them will eventually be pushed past their limit by exactly the monitoring meant to keep them healthy.
The fix was to use the protocol the target actually speaks for automation, a single small network round trip, rather than the one built for a person sitting at a keyboard. The second half of the fix matters too: the probe now needs three consecutive failures before reporting, and reports an unknown state rather than a power loss, so that even when it is wrong it is wrong in a way nobody will act on.
Source: node0 lessons v0.1, lesson 2.6. Sanitized: checklist v0.1, 20260921; addresses, hostnames; voice pass 20260921. Part of oznog.com/node0.