An NVMe controller can die and leave no evidence
- date
- 20260910 and 20260917
- what happened
- An NVMe drive on ceph8 stopped answering writes and reported 'device not ready' to every reset. It survived a PCI remove-and-rescan and a warm reboot of the whole host, and came back only after a full cold power cycle. A week later the same failure on a drive in ceph2 went straight to a cold cycle and recovered identically. Both drives' own SMART counters then read as if nothing had happened.
- what it cost
- A downed OSD each time, with twelve placement groups undersized and eight degraded on the first event, plus the diagnostic time spent re-proving that a warm reboot does not help.
- what changed
- A monitoring rule alerts on an NVMe disappearing from the metrics scrape, because a wedged controller stops reporting rather than reporting a fault. The standing rule became that a second wedge on the same drive means replace it, not power-cycle it again.
- the check now
- An alert on any NVMe vanishing from the scrape, and a per-drive watch with a stated clearing criterion in the inventory, since the drive itself keeps no record.
An NVMe controller can wedge so completely that the operating system still sees the device node while every command to it returns as though the controller has stopped existing. On 20260910 one did exactly that on a storage host: writes stopped, every reset attempt came back “device not ready”, a PCI remove-and-rescan did nothing, and a warm reboot of the entire host did nothing either. Only removing power from the drive brought it back.
The unsettling part is what happened next. The drive kept no internal record of the event at all. Its SMART counters after recovery read exactly as they had before, so a clean post-recovery health check is not reassurance, it is silence. Nothing on the drive proves the wedge happened; only the log a person or an agent wrote down at the time does.
On 20260917 the identical failure appeared on a different drive in a different host. The warm reboot was not tried the second time, the cold cycle was done first, and recovery was identical. Two hosts, two drives, one procedure. That is the strongest evidence this fleet has that a cold power cycle, not a warm reset, is the only real fix for this failure class. It echoes what the CPU IERR hangs taught earlier (lesson 1.4). The rule is scoped to those two signatures, and it runs through the storage cluster’s own drain and shutdown procedure. It is not general advice to power-cycle things.
Monitoring had to change shape for it. A wedged controller does not report a bad value, it stops producing metrics entirely, so the rule that catches it alerts on the disappearance of a series rather than on its contents. Both drives were then put on an explicit watch with a stated clearing criterion in powered-on hours, because with no counter to read, the only way to close a fault is to say in advance what “clean” would look like.
What made a cold power cycle an acceptable cure at all is that the host was one of nine in a cluster. Ceph drains the host’s disks, keeps serving from the other eight while it is dark, and heals when it returns; the same is true of a hypervisor node in the Proxmox cluster, whose guests move before the power goes. On a single box the same fault is an outage and the same fix is a gamble taken with the service down. That is the quiet argument for clustered storage and compute even at a small site: not raw capacity, but the ability to take any one machine cold, for a reboot, a firmware change or a controller that will not answer, without the rest of the site noticing. The drain-and-shutdown procedure the rule runs through exists so that this is a routine rather than a risk.
Source: node0 lessons v0.1, lesson 1.11. Sanitized: checklist v0.1, 20260921; NVMe serials, record ids; voice pass 20260921. Part of oznog.com/node0.