A shutdown that works perfectly and still leaves the fleet dark forever
- date
- 20260910 to 20260913
- what happened
- A clean NUT-triggered shutdown leaves every Supermicro chassis in soft-off. All 14 were set to restore power on Last State, which after a clean shutdown means restoring off. Fixing that uncovered a second trap: on the X10DRU-i+ boards the BIOS enum value 01 means Power On, and on the X10DRH board that runs the 218 TiB array, 01 means Stay Off and 00 means Power On.
- what it cost
- Nothing materialised as an outage. It was found by design review before any unattended power return, but the shutdown design had already been declared finished once, on the basis that shutdown alone was proven.
- what changed
- Per-board-family gold BIOS configuration files, with the enum mapped correctly per family, were built and applied without a reboot to all 14 chassis on 20260913. A companion decision made the NUT sequence command the UPS to cut and restore its own output, so that mains returning is a real AC loss and return the BIOS setting can react to.
- the check now
- A power-return check script, and the knowledge that the enum reads back as the pre-change running value until each host's next boot, which means a real reboot is the only proof.
Every server board has a BIOS setting for what to do when AC power returns: stay off, power on, or restore the last state. Last State is the common default and it sounds like the safe answer. It is not, if your shutdown design works.
A clean shutdown leaves a chassis in soft-off. Last State faithfully restores exactly that. So a textbook UPS-triggered shutdown, executed perfectly across the fleet, guarantees the fleet stays dark when mains returns, with no error anywhere, until someone walks to each machine. Node0 had already treated its shutdown design as finished, proven by successful shutdowns, before anyone asked the other half of the question. Testing “did it stop safely” is easy. Testing “will it come back” needs an actual power interruption, which is why it gets skipped.
The second trap is nastier and specific to mixed fleets. The enum value that means Power On is not portable across board families from the same vendor. On most of Node0’s boards, 01 is Power On. On the one that runs the 218 TiB storage array, 01 is Stay Off and 00 is Power On. Applying the majority family’s value to that host would have silently guaranteed the one machine you most want back never comes back, and the setting reads back as correct until the next boot.
Two rules. Never assume a BIOS setting name or enum value means the same thing on two board families; dump and compare each family’s running configuration. And make the recovery genuinely testable. Node0’s shutdown sequence now tells the UPS to cut and restore its own output, so that mains returning is a real AC loss and return event rather than a UPS that quietly stayed online the whole time.
Source: node0 lessons v0.1, lesson 5.11. Sanitized: checklist v0.1, 20260921; hostnames; voice pass 20260921. Part of oznog.com/node0.