A drive can be physically gone for nine minutes and every health check stays green

The cluster is built the way most performance-minded Ceph clusters are built. Each storage daemon keeps its bulk object data on a spinning disk and its metadata database on a fast solid-state device. That split has a consequence nobody had connected to the earlier 38-hour blind disk. The routine health signals a storage daemon emits, its own up or down state, its heartbeat with its peers, the completion of a shallow scrub, are all satisfied by the metadata. Pull the platter out and the metadata is untouched. The daemon carries on reporting itself healthy, and a shallow scrub of the affected placement groups completes cleanly, because a shallow scrub validates metadata rather than object data.

The team proved this rather than reasoning about it. They pulled a healthy disk out of a live cluster and timed the silence. Nine minutes in, every check was still green. The fix does not ask the storage software for its opinion at all. It follows the daemon’s data device symlink down through the logical volume to the physical device node and checks that the node still exists, reads the drive enclosure’s own slot-presence status, and reads the SAS link error counters, none of which requires issuing a single read to the disk.

Any system that separates data from metadata for speed has this trap, and it is not limited to Ceph: database engines with separate write-ahead logs, caching tiers, and tiered filesystems all get to look healthy from the fast device while the slow one is missing. The durable fix is a check written underneath the software’s own reporting, at the layer where the hardware either exists or does not.