Skip to content
Oznog

1.5 · hardware failures and rmas · after redaction

Identity lives on the drive or the card, never the board

date
20260905 to 20260919
what happened
A Raspberry Pi 5 was condemned for a bootloader fault; its NVMe carrier and drives moved to a spare board, which became the same node again with no reinstall. Ten days later a ClusterHAT controller moved from a Pi 4B to a 3B+ by moving only the SD card, and four days after that two Pi Zero workers were swapped the same way.
what it cost
About a day of cross-testing to separate a failing NVMe carrier from either Pi board. The later swaps cost almost nothing, because the lesson was already understood.
what changed
The working assumption for any single-board computer became that the board is disposable and the storage is the identity. Inventory records for the individual Pi Zero boards were only added after the last swap, because until then nobody could say which physical board had gone where.
the check now
On any single-board-computer fault, move the storage before ordering a replacement board, and verify the swap by node identity, not by which physical unit is racked.

Twice in two weeks, on two unrelated families of single-board computer, a hardware fault that should have cost a reinstall cost nothing but a physical swap.

The first was a Raspberry Pi 5 with an I2C bootloader fault. Its NVMe carrier and both drives moved to a healthy spare board, and the spare simply became the same node: same hostname, same address, same keys, no reinstall and no configuration change, because a Pi’s identity lives on its storage and the board underneath is stateless silicon. The carrier itself was the harder part of that job. It was failing too. The only way to prove the carrier was at fault, rather than either board, was to cross-test it against a second Pi with a different ribbon cable, which took about a day.

The same insight paid twice more on a different rig, a four-node ClusterHAT teaching cluster. On 20260915 its controller moved from a Raspberry Pi 4B to a 3B+ by moving one SD card, and on 20260919 two of its Pi Zero workers were replaced the same way. In every case the hostname, address and network identity followed the storage, with no reconfiguration anywhere.

The gap this exposed is record-keeping, not technique. Nobody had written down which physical board sat in which slot, so the provenance of the first ClusterHAT swap had to be reconstructed from memory rather than read from the inventory. Per-board inventory entries exist now, added after the last swap, which is later than they should have been.

Source: node0 lessons v0.1, lesson 1.5. Sanitized: checklist v0.1, 20260921; board serials, MAC addresses; voice pass 20260921. Part of oznog.com/node0.