Give storage time to prove itself

Storage fails early, and it fails more than anything else. On Node0 that was not a suspicion but a count. Over the three weeks from bring-up on 20260901, the hardware theme’s incidents split cleanly: two boards, three network cards, a transfer switch and a KVM on one side, and the drives and their controllers on the other, with more failures than all of those together. Three of about 80 refurbished drives never spun up. One of three RAID cards was dead. An 18 TB drive in the hub’s main pool died abruptly at 297 power-on hours, in the middle of a scrub, going from a clean history to unable to read sector 0 (1.14). A SATA SSD passed a health check at rest twice and failed in service twice, and a 12 TB drive’s own self-assessment read PASSED throughout 12,232 reallocated sectors (1.17). Two NVMe controllers wedged a week apart, on different hosts, and each came back only from a full cold power cycle, with counters that afterwards read as if nothing had happened (1.11). A dirty bay connector made two good drives look bad in turn, and cost a six-hour drain-and-rebuild that was never about a drive (1.10).

The rule that came out of it is about time. New storage is not trusted with the only copy of anything until it has run under load long enough to fail. In practice that is four habits:

• Burn a drive in before it joins a pool. A sustained write pass and a read-back, long enough to find the drive that dies in its first hundred hours, because that is when they die. A health check at rest is not this; the SSD above passed two of them.
• Scrub a pool before you depend on it. The scrub is what found the 297-hour drive, and the scrub is also what proved the pool had absorbed it: the remaining 52.7 TB finished with zero repairs.
• Reproduce a suspect part under the load that exposes it, rather than checking it at rest (rule 4), and clean the bay before condemning the drive (1.10).
• Collect the counters that matter, per serial, from the first day. The drives that make up most of the Ceph fleet keep their worst numbers outside the standard health table (1.9); a collector that reads the right table, keyed on the serial and not the device letter, is what turns an early failure into a warning instead of a surprise.

What made the count survivable was that none of the early failures ever hit the only copy. The tiers were built with parity to spare, the hub’s groups run three-disk parity, and the first weeks of a new tier were treated as the test they turned out to be. A builder with less redundancy has the same choice in a different form: give the drives their burn-in on the bench, and give the tier its first weeks with a second copy elsewhere.