A raidz3 absorbed an 18 TB drive's sudden death without losing a byte
- date
- 20260915
- what happened
- An 18 TB Seagate Exos drive in unraid1's main pool failed abruptly mid-scrub at 297 power-on hours, going from a clean SMART history to unable to read sector 0 and unresponsive to SMART entirely. ZFS faulted it out of its raidz3 group after 181 read and 151 write errors.
- what it cost
- One drive lost at under 300 hours of service. No data was lost. Two parity disks were still standing, and the scrub that found the failure finished the remaining 52.7 TB with zero repairs.
- what changed
- The replacement was addressed by its ZFS GUID rather than its device letter, because the letter had already moved twice during the fault. A standard claim document was written the same day, to a fixed place where the next person will find it.
- the check now
- Reseat-test any drive before condemning it, to separate the drive from its bay, cable and backplane, and replace by GUID whenever a fault has already caused more than one device-letter reassignment.
Two parity disks exist so a single drive’s death can pass with no drama.
On 20260915, during a routine scrub (the pass that reads every block in a ZFS pool and repairs what does not match its checksum), an 18 TB drive with 297 hours of service and a clean history went from healthy to unreadable. It had not degraded. It could not read sector 0 at all, and it stopped answering SMART queries entirely. ZFS counted 181 read and 151 write errors and faulted it out of its raidz3 group, which is a layout that keeps three parity disks and can therefore lose three members before any data is at risk.
Nothing downstream noticed. The pool kept serving, the scrub carried on and completed the remaining 52.7 TB with zero repairs, and a cold spare rebuilt into the empty slot.
Two small pieces of the handling are the parts worth copying. The first is a reseat test. Pulling the drive and putting it back moved its address on the storage bus, which proved the bay, the cable and the backplane port were all fine and left the drive as the only suspect, before a claim was written. It costs two minutes and it removes three components from the list.
The second is addressing. During the fault, the drive’s Linux device letter had already been reassigned twice, so the replacement was identified to ZFS by its pool GUID instead. A letter is a label the kernel assigns in the order things are found. A GUID is the disk. When a fault has already shuffled the letters once, anything keyed to them is a coin toss.
Source: node0 lessons v0.1, lesson 1.14. Sanitized: checklist v0.1, 20260921; drive serials, claim file name; voice pass 20260921. Part of oznog.com/node0.