The whole fleet had two minutes to shut down a 218 TiB array
- date
- 20260903 to 20260904
- what happened
- The UPS-aware shutdown design stops counting a power supply only once that supply reports low battery, and the UPS units' low-battery threshold was still at its factory default of 120 seconds. In a real outage every host on a rack, including the 218 TiB Unraid array and the whole Ceph tier, would have had about two minutes to stop cleanly.
- what it cost
- Nothing yet. It was caught during the shutdown design's own review rather than by an outage, but a stop of that array had already been seen to take minutes on its own, so the first real test would very likely have produced an unclean shutdown on the storage tier.
- what changed
- The low-battery threshold was raised from 120 seconds to 900 seconds on all four UPS units. Ceph and Proxmox were moved from per-node independent shutdown to one orchestrated stop, since a single OSD host disappearing mid-event leaves a degraded cluster still being written to.
- the check now
- The low-runtime setting reads 900 seconds on all four units, confirmed live. Nothing in the current client set takes near that long to stop; the storage tier was measured at 114 seconds, tested empty.
NUT is the Network UPS Tools suite, and its monitoring daemon does not think in minutes at all. It watches whether each UPS reports itself online or on battery, and it removes a UPS from a host’s count of working supplies only at the moment that UPS declares low battery. An OSD is a Ceph object storage daemon, one per disk; there are a lot of them, and they do not stop instantly.
Setting MINSUPPLIES to 1 on a host with two feeds is an elegant design. The host rides out an entire outage on one surviving feed and never notices the other one died. The consequence, which is easy to miss, is that all of the reaction time your fleet gets is hidden inside a single UPS setting that most deployments never touch. The UPS computes low battery as remaining runtime at the current load, and the factory default here was 120 seconds. Once both feeds of a rack crossed that line, every host on it had two minutes and a five-second final delay to stop cleanly, including a 218 TiB array that had already been observed wedging for minutes during an ordinary stop.
Raising that one number to 900 seconds converted an already-deployed shutdown chain from a scramble into something comfortable, with no change on any client host at all. It was chosen over a more elaborate early-warning design for a specific reason. At the time, power return was not yet built, so an unwanted shutdown was judged more costly than a slower reaction.
If you deploy NUT with MINSUPPLIES, treat the UPS’s own low-battery threshold as your actual shutdown deadline, and size it against the slowest thing you run.
Source: node0 lessons v0.1, lesson 5.5. Sanitized: checklist v0.1, 20260921; hostnames, watt figures; voice pass 20260921. Part of oznog.com/node0.