A five-day-old battery warning fenced a hypervisor
- date
- 20260915
- what happened
- prox1's MegaRAID controller, which carries the host's root pool as two passthrough drives, had logged a failed cache battery once a day for five days with no alert anywhere. The controller firmware then faulted hard, root I/O stalled, the cluster heartbeat dropped, and the cluster's high-availability watchdog fenced and reset the node.
- what it cost
- An unplanned fence and reboot, several hours of investigation, and a near miss. Blacklisting the storage driver was nearly the chosen fix; it would have left the machine unbootable, since both root pool drives sit behind that controller.
- what changed
- The failed battery, a supercapacitor protecting a write-back cache that does not exist on these nodes, was disconnected on all four hypervisors by decision, ending a continuous relearn-and-fail loop on every one. Battery monitoring moved from a log-line rule to a rule that polls the controller directly.
- the check now
- Poll every node's battery health directly and alert on any high-availability fence event, because a failing component can log its own fault for five days without tripping anything else.
A component can fail completely and keep announcing it once a day for five days without producing a single alert, if the only thing watching is a rule built to match one specific log line.
On 20260915 a hypervisor’s RAID controller stopped being patient. Its firmware faulted, input and output to the host’s root pool stalled, the cluster heartbeat went with it, and the cluster’s high-availability watchdog fenced the node and reset it. That part worked as designed: no data was lost, the root pool came back with zero errors, and the guests that had been running were restarted on other nodes automatically.
The investigation found two things worse than the original fault. The first is that the warning had been in the logs for five days, every day, unwatched. The second is that the failed component is a supercapacitor whose job is to protect a write-back cache. These nodes run no virtual drives at all, only passthrough, so the cache it protects does not exist. Every one of the four hypervisors had the same failed battery. Only the one that happened to crash had ever phrased it in the exact words the monitoring rule was looking for.
The investigation also came close to a fix that would have been worse than the fault. Blacklisting the storage driver was on the table until someone traced which drives actually sit behind that controller on that specific machine, and found both halves of the root pool there.
The cheap fix was to unplug a part that was protecting nothing. The real fix was structural: stop reading what a component chooses to say and ask it directly, on a schedule, whether it agrees.
Source: node0 lessons v0.1, lesson 1.13. Sanitized: checklist v0.1, 20260921; guest identifiers, controller serials; voice pass 20260921. Part of oznog.com/node0.