CARP split-brain that was two bad cables, not bad config
- date
- 20260817
- what happened
- Bringing up the OPNsense firewall pair fw1/fw2 on the bench, both firewalls claimed CARP master on two of three shared virtual addresses at once. Authentication error counters climbed on both nodes, which pointed at a mismatched shared key even though the stored keys hashed identically.
- what it cost
- The better part of a day of protocol-level theories (rekeying, recreating the virtual addresses, repeated reconfigure passes) before the physical layer was suspected.
- what changed
- Moving both interconnect legs off a temporary unmanaged switch and onto the production 10GbE switch converged all three virtual addresses with no configuration change. A standing check and a diagnostic habit came out of it. Force a normally silent peer to transmit on a known-good link before concluding it cannot transmit.
- the check now
- After any change to a virtual address, a shared key, or cabling, confirm exactly one master per virtual address from a third host, never from either firewall's own status page.
A pair of firewalls that are supposed to fail over into each other instead both decided they were in charge at once. That is the one CARP failure (Common Address Redundancy Protocol, the mechanism that lets two firewalls share one gateway address). It looks completely normal from a status page: every shared address answers, every check passes, and there is quietly no redundancy at all. It happened on 20260817, during bench bring-up, on two of the three shared addresses.
The authentication error counters climbed on both nodes. A mismatched shared key produces exactly that symptom, so the counters were chased for hours. They were the wrong thing to trust. A failing cable delivers corrupted frames, and a corrupted frame counts as an authentication failure whether or not the key is right. The one virtual address that worked was the one wired as a direct crossover. The two that failed both ran through a temporary unmanaged 2.5GbE switch, which handled CARP’s multicast asymmetrically. One of the firewalls also had a failing cable on the interface involved. Two independent faults sat in the temporary lab cabling.
The fix was to move a cable. Getting there took forcing a firewall to become master on a link with no switch in the path. A CARP backup normally only listens, so it had never once had to speak. That single test is what separated “this node cannot send multicast” from “this node’s cable cannot carry it”.
The lesson generalises well past CARP. Any protocol with a normally silent role, a backup, a standby, a replica, can hide a broken transmit path indefinitely. Nothing ever asks it to prove it can send. When several protocol symptoms sit on one interface and an identically configured interface is fine, suspect the physical link before the protocol.
Source: node0 lessons v0.1, lesson 4.1. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.