Blown fuses are invisible to the eye, and one already cost a transfer switch
- date
- 20260820
- what happened
- Each of the four rack PDUs carries four branch fuses, sixteen across the fleet, and a blown one is physically identical to a good one. This was discovered from a real incident. A transfer switch failure shorted and blew a fuse, localised only by reading the PDU's own status display rather than by inspection.
- what it cost
- One transfer switch destroyed by the underlying short and replaced from spares, plus the time spent narrowing down which of sixteen visually identical fuses had actually failed.
- what changed
- Fuse status is checked through the PDU's interface rather than by eye or meter as a first resort. Spare fuses were increased to eighteen on hand, enough to cover a full sixteen-fuse loss with margin, and the transfer-switch spare policy was raised from one unit to two.
- the check now
- The PDU's own line and outlet status fields distinguish a fused branch from a live one, which means the monitoring stack can alert on it directly rather than requiring a manual look.
A small, concrete fact for anyone running rack PDUs with internal branch fuses. You cannot tell a blown fuse from a good one by looking at it. There is no discoloured element, no visible break, nothing. Sixteen of them across four PDUs look exactly alike whether they are carrying current or not.
Node0 learned this the expensive way rather than the theoretical way. A transfer switch failed and shorted, taking a branch fuse with it. The transfer switch itself was destroyed and replaced from spares. Finding which fuse had gone was the part that could have burned the most time, because the instinct is to open the PDU and look, and looking tells you nothing.
The working answer was already in the rack. The PDU reports its own branch status, and a blown branch reads distinctly there, as a specific status code on the interface, even though the physical part gives nothing away. That is now the first check rather than the last, and because it is a field the PDU already exposes, the monitoring stack can watch it continuously instead of waiting for someone to go and look.
Two habits follow. Learn a device’s own fault-reporting interface before an incident forces you to discover it, because the middle of an outage is a bad time to find out which field means what. And stock spares generously for this class of part. One event took out a transfer switch and a fuse at the same time. That single fault is why the fuse stock went to eighteen and the transfer switch spare policy went from one unit to two. A single fault can consume more than one thing at once.
Source: node0 lessons v0.1, lesson 5.19. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.
