The route to the power controllers ran through the thing they had to survive
- date
- 20260913
- what happened
- Tracing how the two servers that manage the UPS network cards actually reach them found that both routed through the pair of firewalls by default rather than directly. If the firewalls halted before the last UPS had received its output-cut command, that command would never be sent.
- what it cost
- Nothing yet. It was found by deliberately tracing the route rather than by an incident, but the margin that hid it, the firewalls' own small UPS units running for well over two hours at their light load against a plan built around 15-minute deadlines, made it an easy blind spot.
- what changed
- Both management servers were given a second, tagged network path directly onto the isolated power-management network, so polling and the output-cut command never cross the firewalls at all. The firewalls' dependency on a full stop before their own last step is documented as the fallback.
- the check now
- The UPS network cards' own access-control lists now admit the management servers from their new direct path, with the old firewall-routed path kept only as a fallback.
The whole power-return design rests on one command reaching each UPS during an outage: cut your output, and bring it back when mains returns. Somebody asked the unglamorous question of what that command actually crosses on its way, and ran the route lookup rather than assuming.
Both of the servers that manage the UPS network cards reached them through the firewall pair, by ordinary default routing. Nothing was misconfigured. It was simply the path the routing table produced, and no one had looked.
The consequence only appears if you follow the sequence to its end. The firewalls are themselves powered by small UPS units subject to the same outage. If they halt before the last UPS has been told to cut its output, that UPS never gets told. Every rack is then left soft-off with its UPS still online and fully powered, which no script can recover from, because the thing that would send the command is off. That is precisely the failure the design existed to prevent, produced by the one host class the design had not accounted for.
What hid it was a comfortable margin. The firewalls’ own units run for well over two hours at their light load, against a plan built around 15-minute rack deadlines. Sequencing around that margin was considered and rejected, because it would have made the fix a property of the firewalls’ current load rather than a structural guarantee. Instead both servers were given a second, tagged path directly onto the same isolated network the power hardware already lives on, so the circularity is gone rather than managed.
The principle is plain. No control channel your recovery depends on may itself depend on the thing you are recovering from. Trace the route your control commands take. Do not assume it.
Source: node0 lessons v0.1, lesson 5.14. Sanitized: checklist v0.1, 20260921; addresses, VLAN numbers; voice pass 20260921. Part of oznog.com/node0.