Board damage an elimination tree cannot see: ceph7's three motherboards

One host needed three motherboards in nine days, and each swap taught something different.

The first board’s SAS path (SAS is the disk interface that carries this chassis’s twelve drive bays) was simply dead. The elimination tree ran nearly a full day: a BIOS settings diff against a known-good node, the fleet’s gold bifurcation settings applied, two different host bus adapters tried, a riser swapped in from a sibling chassis, controller and PCI resets, a fresh CMOS cell, a CPU swap between sockets, and a BIOS reflash the board refused. None of it found anything, because the fault was a physically damaged connector on the riser slot, and that was visible in seconds once the board was out of the chassis and under bench light.

The second board fixed the SAS fault and brought its own: one PCIe root port electrically dead, so seven of eight NVMe drives enumerated instead of eight. The third board, fitted 20260911, fixed the dead lane and quietly broke an assumption the fleet had just spent a week hardening into policy, that every board here carries a previous owner’s signed BIOS that cannot be updated. It did not. It was retail and updatable, the first confirmed exception.

Two habits came out of this. Look at the contacts before running the tree, because the bench sees what the rack cannot. And check each replacement part on its own terms. A fleet built from used hardware is never as uniform as its purchase order implies, and assuming a replacement matches its predecessor costs a week when it does not.

A third habit is about who does what. This diagnosis was run by an agent, and an agent iterates on configuration and software in seconds; every step here that needed hands cost a person’s time and, around it, a host outage: drain the host, power it down, pull the board, do the work, refit, bring it back, verify. Ten iterations of that shape is a week, and it was. So an agent working a hardware fault exhausts what software can rule out first, then groups every request for hands into one visit, ordered by what each would eliminate, with the parts, the cables and the read-back for each step written down before the person walks in. Rapid iteration is the agent’s strength on a keyboard and its most expensive habit on a rack.