Skip to content
Oznog

1.17 · hardware failures and rmas · after redaction

Proving an RMA: what Node0 learned by doing it

date
20260904 to 20260916 (the resulting policy is ongoing)
what happened
Early claims taught Node0 that a health check performed at rest proves nothing: a SATA SSD passed SMART twice and failed in service twice, and a 12 TB drive's own SMART self-assessment read PASSED throughout 12,232 reallocated sectors and the data inconsistency it eventually caused. An inventory audit on 20260916 then found five already-claimed, already-dead drives sitting on the spares shelf as available stock.
what it cost
Two drives cleared as healthy and returned to service, twice each, before they failed for real, and five phantom spares that would have been handed out as good stock.
what changed
A two-tier test tool, a short pass for a spare and a longer repeatable pass for a suspect, with fixed per-serial evidence capture. The inventory grew a three-axis model, parent device, lifecycle status and a dated disposition, so the question of what happened to a given serial has a permanent answer.
the check now
Never clear a drive on an at-rest SMART pass alone, and check a part's disposition field before trusting anything on a spares shelf, rather than whether it is physically present.

Every return-merchandise process eventually meets the same trap. A manufacturer’s health check is written to describe the drive, not to predict whether it will survive the next hour of real use. A drive can pass it convincingly right up until it fails in production for the second time. Two did here: an SSD that passed SMART twice and failed in service twice, and a 12 TB disk whose own self-assessment read PASSED the entire way through 12,232 reallocated sectors and the storage-level inconsistency those eventually caused.

The fix that stuck was to stop asking the drive for its opinion and start measuring the condition the suspected fault actually needs. For SSDs, the tell turned out to be power-cycle count read against power-on hours. The condemned drive’s controller was resetting itself roughly once per powered minute, against well under once per ten hours on its healthy siblings. That difference is unmistakable while barely disturbing a health check. For hard drives it is a load-reproducing burn-in pass rather than an idle read. Both live in one tool with two tiers, a short pass for a spare going back on the shelf and a longer repeatable pass for a suspect, each capturing its evidence per serial number.

The second half of the lesson is organisational, and it surfaced on 20260916 when an inventory audit found five drives on the spares shelf that had already been claimed and were already dead, reading as available stock, because the records had never distinguished “physically present” from “usable”. A spares shelf that only grows and never gets audited will end up holding dead stock that looks exactly like good stock. The repair was a data model, not more careful handling. Every part now carries a dated disposition alongside its status, including the deliberate decision to run drives with no recorded warranty until they fail rather than file a speculative claim.

Source: node0 lessons v0.1, lesson 1.17. Sanitized: checklist v0.1, 20260921; drive serials, claim file names; voice pass 20260921. Part of oznog.com/node0.