Three X540 NICs die in one week, three different reasons

The same NIC chip failed on three different hosts inside four days, and treating that as one story would have been a mistake. It was handling damage on the first, a silent speed downgrade on the second, and a thermal design problem on the third.

The firewall’s add-in card lost carrier on both ports twice, once after physical handling and once 360 seconds after an otherwise untouched reboot, and was replaced after a third drop. The last one is the clean case: nobody touched it, and it still let go.

The middle case is the instructive one. A port that should have run at 10 Gb negotiated 100 Mb and reported itself up the entire time. Nothing watching link state could see anything wrong, because nothing was wrong with the link, only with its speed. What caught it was Linux’s own 802.3ad bonding refusing to aggregate two slaves running at different speeds, which showed up as two different LACP aggregator IDs on one bond. Without that, a host quietly running at half bandwidth and zero redundancy would have kept working, just more slowly, indefinitely.

The third was thermal. One chassis’s riser ran its NIC sensor 15 to 20 degrees C hotter than identical siblings under the same fan duty. A heatsink re-pad did not fix it and the riser was replaced.

The lesson is not about X540s. It is that “link up” is a much weaker claim than it sounds, and that identical symptoms across nominally identical hardware are not evidence of one shared cause.