Ceph hosts hang with no OS-side trace

On the morning of 20260906 two storage hosts hung hard, hours apart. Each had a processor IERR (an internal error the CPU raises when it cannot continue) in the log kept by its board management controller, the small always-on computer that watches a server independently of the operating system. Both machines were powered, both were unresponsive, and neither had anything left to say: their consoles were blank and their own logs had stopped being written. Ceph kept serving from whatever survived, two of three monitors and a fraction of its OSDs (an OSD is the daemon that owns one disk), for about ten hours, and nothing paged.

The only external tell was on the switch. Each hung host was emitting roughly 600 Ethernet PAUSE frames per second on every port. PAUSE is flow control: a network card whose receive ring nobody is draining asks its peer to stop sending, and a card belonging to a dead operating system asks forever. That is the one signal a truly dead host can still produce, and it arrives somewhere other than the host.

The fix for the hang was mechanical. A warm reset re-triggered the identical fault on both machines within minutes; only removing power cleared it. The fix for the silence was structural: poll the management controller’s event log across the fleet, and treat PAUSE-frame growth on an access port as a sick server rather than a sick network. The same cold-boot-only pattern turned up later on a completely different component, an NVMe controller (lesson 1.11), which is the best evidence available that it generalises.