A 208/240 V mismatch that looked like a bad transfer switch
- date
- 20260824
- what happened
- Two of the four APC Smart-UPS SRT 5000 units were already set to 240 V output; the other two were still at 208 V. A rack's automatic transfer switch read its preferred source as under-voltage, parked itself on the other source, and looked like a failing transfer switch.
- what it cost
- The diagnosis time to trace a transfer-switch symptom back to a UPS setting two devices upstream, plus a firmware-level configuration push on all four units.
- what changed
- All four UPS units were standardised to 240 V output, and the procedure was written down, including the fact that SRT output voltage is changed only by a config-file push made with the output off.
- the check now
- The transfer switch's own selected-source field, and the matching Prometheus metric, both read correctly once the two sources agree on voltage; a future mismatch produces the same Source Not Ok signature.
An automatic transfer switch, or ATS, is a device with two power inputs and one output. It picks a healthy source and switches to the other one if the first goes away. It decides “healthy” by measuring the input against a voltage window. That is the whole mechanism, and it is why this failure is so convincing.
Node0’s four APC Smart-UPS SRT 5000 units are all rated for both 208 V and 240 V output. Two had been left at one setting and two at the other, which nobody noticed because each unit worked perfectly on its own. The rack fed by one of each saw its preferred input sitting outside the window the transfer switch expected, reported that source as not OK, and stayed parked on the other one. Everything downstream kept running, so the only visible symptom was a transfer switch that appeared to have lost a source, on a rack where both sources were live and correct.
The fix took about five minutes per unit once it was found, and it is worth knowing in advance. On this UPS family, the output voltage is not something you can change from the command line while the unit is running. It is a configuration file you push with the output switched off, then switch back on.
The general shape outlives the vendor. Any redundant-source device that judges a source against a voltage window can be fooled by two genuinely working sources that simply disagree about nominal voltage, and it will report that disagreement as a fault on the good side rather than as a mismatch. Before wiring an ATS to two supplies, read each supply’s actual output voltage, not its rating.
Source: node0 lessons v0.1, lesson 5.1. Sanitized: checklist v0.1, 20260921; addresses, tool path, an SNMP exposure; voice pass 20260921. Part of oznog.com/node0.