Kea DHCP high availability: five defects before it actually failed over
- date
- 20260817 (lab, four defects) and 20260827 (production, a fifth)
- what happened
- Building DHCP failover between the two OPNsense firewalls with Kea's hot-standby model, five separate defects each produced a service that reported no error a casual check would catch: a competing daemon holding the DHCP server port, a control channel colliding with the failover channel it was meant to enable, a socket mode that would have deafened the server to broadcast requests, an autocollect option substituting a node's own address for the shared one in the gateway it handed clients, and a configuration-sync process copying a setting that is only correct when it differs per node.
- what it cost
- Each defect independently produced a fully enabled but non-functional pairing. The production defect left one network's DHCP with no real redundancy for about a day (20260827 to 20260828). The single affected client tolerated it without an outage.
- what changed
- Kea was removed from the firewall pair's configuration-sync list entirely, making its per-node settings genuinely per-node. Failover was then proven end to end on real hardware on the evening of 20260828, with a client rebinding the same lease address from the newly promoted node.
- the check now
- Read the failover state machine's own log lines rather than trusting a GUI enabled flag or a running process, and check the effective value of every address an HA pair emits, not the value that was entered.
Setting up redundant DHCP sounds like flipping one switch. Enable high availability on both firewalls, done. It took five separate, independently silent defects before Kea’s hot-standby model actually did what its name promised, and every one of them looked fine from the ordinary places anyone would check. The interface showed enabled. A process was running. A status page reported no error.
Each defect was a default that quietly did the wrong thing in a failover context specifically. The firewall’s own lightweight DHCP daemon still held the port Kea needed, and the log message never named it as the cause. The control agent, which the firewall needs to write the failover hook into Kea’s configuration at all, was pointed at the same port as the hook itself. A spurious warning tempted a change of socket mode that would have made the server unable to hear broadcast discovery packets entirely. An autocollect convenience replaced the pair’s shared gateway address with one firewall’s own, so a failover would have handed clients a route to a dead firewall. And ten days later, in production, the config-sync process copied the setting by which each node names itself, so the backup could not find itself in its own peer list.
The general shape is worth carrying elsewhere. In any active or passive pair, ask of every emitted value, not just every enabled feature, whether it is the shared identity or one member’s own. Then verify by reading the service’s own state machine, the transitions and the enable and disable events it logs, rather than a dashboard’s summary of them.
Source: node0 lessons v0.1, lesson 4.4. Sanitized: checklist v0.1, 20260921; subnets, client reservation detail; voice pass 20260921. Part of oznog.com/node0.