The hard drives were hiding their worst counters, and a reboot's letter shuffle paged seven false alarms

Two unrelated mistakes compounded into one class of failure on the same day. The first: on the fleet’s most common hard drive model, the SMART attribute table that everyone reads simply does not carry a field for read retries or uncorrectable reads. Those counters exist, in a separate ATA device-statistics log that most tooling never asks for, and one drive in the fleet turned out to be carrying hundreds of thousands of hidden read retries behind an attribute table that looked spotless. The data had always been there; nothing had ever queried it.

The second: the growth-based rules meant to catch a counter climbing were keyed to the drive’s operating-system device path. A set of unrelated reboots reshuffled which path mapped to which physical drive. Every rule then compared one drive’s counter to a different drive’s history, and seven alarms fired for drives whose condition had not changed at all.

The fix to the first was to collect more data. The fix to the second is a design principle worth generalising well past disks. Whenever a fleet alert can be triggered by identity churn rather than a genuine change of state, key it on the object’s own permanent identity, not on a name the operating system is free to reassign at boot. Then compare against a value that only ever ratchets upward. A transient dip, a letter swap, a collector glitch, or a momentary zero reading can then never read as a recovery, and a transient recovery can never manufacture a false page. Only sustained growth against a retained high-water mark is left to alert on, which is the only thing you wanted to know about in the first place.