The virtual-MAC fault on the circuit: a two-day diagnosis with the provider
- date
- 20260818 to 20260820
- what happened
- On going live on the site's static address block, any public address carried by the firewall pair's shared virtual MAC address stopped passing outbound traffic after a failover and stayed dead for hours, while an ordinary interface address on the same wire worked fine. It looked exactly like a misconfiguration on our side.
- what it cost
- Two days of measurement, including a written support packet on 20260819 and a live diagnosis session that ruled out roughly a dozen wrong theories, were needed before the cause was pinned. The provider's edge equipment bound each public address to the first MAC address that spoke from it, and a virtual MAC, which by design never changes, was never relearned.
- what changed
- The provider's own engineer rebuilt the circuit on 20260820 as a plain Layer 2 handoff with the address-to-MAC binding removed. Failover over the uplink started working the same afternoon, proven on the wire: abrupt uplink loss gave a 4.1 second failover with 100% of packets replied, and a full chassis reboot also replied to 100% (measured 20260820).
- the check now
- Before committing a CARP or VRRP pair to a new provider circuit, test that outbound traffic sourced from the virtual MAC survives a forced failover, on the wire with a packet capture, not by trusting a ping. Keep a known-good fallback path on a separately established binding until the new design is proven live.
A firewall failover is supposed to be invisible to everyone outside the building. A virtual MAC address, shared by both firewalls, exists precisely so that an internet provider never has to notice which physical box is answering. Node0’s provider noticed anyway. Its edge equipment bound each public address to whichever MAC address last spoke from it, and it never relearned a virtual one automatically. Every address carried by the pair went dark after a failover and stayed dark.
The failure mode was cruel, because it looked identical to a misconfigured virtual address on our side. It cost a wasted day of internal debugging before a careful support packet was written on 20260819 and the provider’s own engineer looked at the circuit. The provider made the actual fix. On 20260820 the circuit was reprovisioned as a plain Layer 2 handoff, with the binding between address and MAC removed entirely.
What made this a two-day diagnosis rather than an open incident was the work done before cutover: reading vendor forums for the exact failure signature, deliberately choosing a virtual router identifier on the uplink that would not collide with the provider’s own protocols, proving that a MAC-spoofing contingency worked as a fallback, and keeping a known-good fallback router on its own already established binding. None of that was wasted. It is what allowed a live circuit to be broken and fixed without the site going down.
The transferable lesson is about shared identities. When a redundancy mechanism depends on a virtual identity that both members present, that identity has to be provably transparent to everything upstream of it. “Provably” means forcing a failover and watching the packets, before anything depends on it.
Source: node0 lessons v0.1, lesson 4.2. Sanitized: checklist v0.1, 20260921; provider name, address block, contacts; voice pass 20260921. Part of oznog.com/node0.