HGST drives hide their worst counters, and a reboot can fake a SMART page
- date
- 20260909 to 20260910
- what happened
- The HGST HUH721212ALE601 drives that make up most of the Ceph HDD fleet carry no SMART attribute 187, so uncorrectable reads and read-recovery attempts never appear in the standard attribute table at all. A fleet-wide survey of all 57 of them, on 20260910, found real outliers nobody could previously see. Separately, growth alerts keyed on a drive's /dev path, so a riser swap and a reboot each reshuffled device letters and paged seven alerts, three of them critical, for drives whose counters had not moved.
- what it cost
- A false-alarm storm of seven pages, three critical, produced by routine maintenance rather than by any drive degrading.
- what changed
- A collector now reads device statistics and the ATA error log fleet-wide on a five and fifteen minute timer, and growth alerts key on retained per-serial high-water marks instead of an absolute count or a device path.
- the check now
- Join any SMART series to its serial number before trusting a growth alert, and read the device statistics pages, not just the attribute table, before declaring one of these drives healthy.
Two blind spots compounded each other inside a single week, 20260909 to 20260910.
The first belongs to the drives. SMART, the self-reporting standard every monitoring tool reads, lets a manufacturer choose which attributes it exposes, and this HGST model does not expose attribute 187, the uncorrectable-read count. Those events are still counted. They live in the drive’s device statistics pages, which almost nothing reads by default. So a drive whose attribute table looks spotless can already be logging uncorrectable reads and read-recovery attempts that nobody is watching. A survey of all 57 of these drives on 20260910 found real outliers that had been invisible until then.
The second belongs to Node0’s own alerting. The growth rules that page when a counter climbs were keyed on the drive’s current Linux device letter, and a device letter is not stable. A riser swap on one storage host and a reboot-then-cold-cycle on another each reshuffled the letters, and the rules compared one drive’s counter against another drive’s history. Seven pages, three of them “failing now”, for drives whose actual numbers had not moved at all.
Neither gap was dangerous alone. Together they were on their way to teaching the person on call that the SMART alerts are noise, which would have been much more expensive than either fault. The repair was to read what the drive actually tracks, on a timer, and to key every growth comparison to a serial number with a retained high-water mark, so that neither a letter swap nor a momentary zero reading can page anyone. The same week seen from the alerting side is lesson 2.8.
Source: node0 lessons v0.1, lesson 1.9. Sanitized: checklist v0.1, 20260921; drive serials; voice pass 20260921. Part of oznog.com/node0.