A management card that lies quietly when two callers ask at once
- date
- 20260901 (found), 20260908 (made worse)
- what happened
- The four APC UPS network cards accept exactly one SSH session at a time. A second, simultaneous login gets no error, only an empty string, indistinguishable from a dead device. Two of Node0's own scheduled jobs collided on this for weeks. On 20260908 a fast loop of sessions across all four cards wedged their SSH stacks entirely.
- what it cost
- Weeks of spurious paging that had to be correlated by hand before the cause turned out to be two of the fleet's own pollers, plus a loss of the management path to all four UPS units for over thirty minutes on 20260908.
- what changed
- The health-check poller retries three times with backoff across the collision window, both jobs take a shared lock around card access, and line endings are stripped consistently so a status string cannot mismatch itself. The standing rule is never to loop over these cards and never to kill a session mid-handshake.
- the check now
- Verified under forced contention: a single-shot poll failed 1 time in 4, the retrying version failed 0 times in 4, and a normal poll still completes in about 5 seconds.
Older embedded management cards often accept one interactive session at a time. That limit is not the problem. The problem is how they refuse the second one. They give no error and no busy message, only an empty response that a script cannot tell apart from a device that is gone.
Node0 ran two scheduled jobs against the same four UPS network cards, a nightly configuration backup and a health check every ten minutes. When their schedules overlapped, one of them got nothing back and the monitoring stack published “UPS unreachable”, then “UPS fine” on the next pass. That pattern repeated for weeks. Nothing external ever touched the cards. The false alarm was entirely self-inflicted, and the shape of it, a device that is up flapping to down and back, is exactly what a real intermittent fault looks like.
The second half is sharper for anyone writing automation against this class of hardware. On 20260908, opening sessions in a fast loop across all four cards, with impatient timeouts that killed some of them mid-handshake rather than letting them log out, wedged every card at once: port 22 open, no login banner at all, a correct password refused, for more than thirty minutes. SNMP polling and UPS output were unaffected the whole time. Only rebooting the cards cleared it.
Two rules follow. Serialise access, one device and one session at a time, and let every session close cleanly rather than force-killing it. And treat a single empty or failed sample as inconclusive rather than as proof of a dead device. Retry with backoff before you alert. A poller that treats one bad sample as truth will publish false alarms forever.
Source: node0 lessons v0.1, lesson 5.4. Sanitized: checklist v0.1, 20260921; addresses, job names; voice pass 20260921. Part of oznog.com/node0.