Why cross-rack switch pairing sounded right and was wrong
- date
- 20260817
- what happened
- An earlier draft of the Arista fabric design paired each rack's switches with a partner in a different rack, reasoning that a host would keep one working connection even if its own rack's switch pair failed entirely. Checked against the actual rack layout, the reasoning did not hold. The routing pair, which holds every gateway address, sat in the same rack as most of the compute and storage anyway.
- what it cost
- Nothing realised. It was caught in design review, before any cabling was bought or run, and it would have required cable runs the existing plant could not support.
- what changed
- The fabric was built with in-rack switch pairs. The resulting single point of failure, that losing the rack holding the routing pair takes down all routing fleet-wide, was written down and explicitly accepted rather than left as an implicit assumption.
- the check now
- When a redundancy design claims a benefit, check what actually shares the protected host's failure domain. A benefit that only helps a component the host does not depend on for its own operation is not a real benefit.
This is a design correction rather than an incident, and it is here because it changed a real decision on 20260817, before that decision became expensive to reverse: a long high-speed cable run between racks that the existing plant could not support anyway.
The earlier reasoning had genuine appeal. Spread a switch pair across two racks, using multi-chassis link aggregation so a host’s two uplinks land on two switches that behave as one. Then a host keeps a working link even if its own rack’s network hardware fails entirely. Written down that way it sounds like more redundancy for the same money.
It assumed that the thing doing the actual routing, the shared gateway addresses every host depends on to reach anything outside its own segment, lived somewhere safe from that failure. It did not. The routing pair sat in the same rack as most of what it served, so a host’s own rack dying was already fatal to it, with or without a surviving cable to a switch elsewhere. The design bought a link that would survive in a scenario where the host it served would not.
The habit worth keeping is to state explicitly which failure domain a redundancy design is protecting against, then check whether the shared infrastructure the design depends on sits inside that same domain. When the answer is that it does, the honest move is the one taken here: build the simpler thing, write the single point of failure down in the runbook, and accept it on purpose rather than discovering later that you had been assuming otherwise.
Source: node0 lessons v0.1, lesson 4.10. Sanitized: checklist v0.1, 20260921; rack and device identifiers; voice pass 20260921. Part of oznog.com/node0.