The Intel chip that eats its own DHCP replies
- date
- 20260828
- what happened
- While debugging why a router on one of the networks the firewall pair serves reported no internet, one of the two Intel network ports on each firewall, the I226-LM, was found to silently drop every outbound UDP packet destined for the DHCP client port, both unicast and broadcast. The drop happens below the driver, so it is invisible to both the packet filter and a capture on the same host. The otherwise identical I226-V port on the same board does not do this.
- what it cost
- The better part of a day, including swapping two switches under suspicion before the network port itself was identified as the cause.
- what changed
- The roles of the firewalls' physical ports were swapped. The affected port now carries the inter-firewall sync link, which never serves DHCP, and the unaffected port carries the network where DHCP is served.
- the check now
- A one-shot test proves it in a single capture: send UDP broadcasts to the DHCP client port and to an arbitrary control port from the same box, and confirm at a neighbouring device that only the control port arrives.
Some hardware faults look exactly like software faults, and this is one of the cleaner examples. An Intel network chip’s manageability filter, present because the chip supports remote management features Node0 never provisioned, ate every DHCP server reply leaving that port. Nothing was visible to the operating system, and nothing was visible to a packet capture run on the same box, because the drop happened in hardware below where either could see it. The sending host shows the frame handed to the driver and then it simply does not exist.
The symptom, from the client side, is a device that asks for an address forever and never gets one, which looks exactly like a client fault or a switch fault. On 20260828 the first two suspects were both switches on the path, and both were innocent. The fix, once found, cost nothing but swapping which logical role used which physical port. The affected port carries a link that never serves DHCP.
The diagnostic worth keeping is more general than this chip. A service can seem to work on the sending host: the process is running, the configuration is right, a capture even shows the frame being handed to the driver, and yet nothing ever arrives. Test whether the frame leaves the interface at all by capturing on a neighbouring device rather than on the sender. A capture on the sender is a record of what the host intended, not of what went onto the wire.
The narrow version of the rule, for anyone with this hardware: never assign a DHCP server or relay to an I226-LM port, and if one is already in service, run the two-port broadcast test before trusting any DHCP troubleshooting done on that segment.
Source: node0 lessons v0.1, lesson 4.7. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.