An agent caused a hardware fault, and it is on the record

The proximate cause was a mismatched or defective controller board. The transfer switch reported itself as a different, lower-voltage product than the chassis and network card around it actually were, and read both of its inputs at about half the real line voltage. Nobody, human or agent, had a reason to doubt that self-report before acting on it. Setting the configured line voltage to match what the unit appeared to be was a reasonable change given the information available, and it was made with an operator’s explicit approval.

It was also wrong, and it could not be undone. With the new setting the controller measured both of its inputs as under-voltage at once, faulted both, and stopped protecting anything. Reverting was refused by the card twice. A management reboot did not help. A full power cycle did not help. The unit was replaced from a cold spare, and a host spent twelve hours on a single unprotected source.

What makes this worth publishing is less the mistake than what happened afterwards. The fault, its timeline, and the fact that an agent’s action caused it are written into the same permanent hardware record every other event on that device goes into, in the same words a human-caused fault would get, with no softer framing. Node0 is run in large part by agents. A record that quietly omits which of them broke something is not a record.

Two things carry over. Verify a device’s self-reported identity against independent evidence, such as a measured reading compared with a known-good sibling, before changing configuration based on it. And keep cold spares for the class of device whose failure removes redundancy rather than service. That policy, raised earlier after a different transfer switch was destroyed by a short, is the only reason this was a same-day recovery.

There is a third, and it is the expensive one. The unit had arrived with a controller that did not match its chassis, which is a defect a vendor would have taken back. Nobody had recorded which vendor it came from. The site’s stance for most of its equipment was already to record the serial and the seller on arrival and to test at once, and this switch was one of the few that slipped through it; the result is about $550 that cannot be recovered, because there is nowhere to attribute it. That made the stance a rule (rule 26). A second-hand part is a claim waiting to be made, and the claim needs three things written down before the box is even opened: the serial, the vendor, and the date it was tested.