A NAT rule with a misspelled field name took public services down for six minutes
- date
- 20260906
- what happened
- A firewall address-translation rule was created through the OPNsense API using field names that were subtly wrong, a flat pair of names where the API expects a nested structure. The API accepted the request, returned success, and stored a rule with no destination and no port at all, which the packet filter read as: redirect all inbound web traffic to one internal host. It applied immediately.
- what it cost
- Roughly six minutes of every public-facing web service being unreachable, redirected instead to a single internal host. An unrelated administrative connection to the external monitoring machine landed on the wrong internal host during the same window. The pair's configuration sync also carried the broken rule to the second firewall before it was caught.
- what changed
- A standing rule for creating any firewall or translation rule through the API: create it disabled, read the stored rule back and verify every field against intent, then enable it and re-apply, then confirm the live packet-filter state shows the specific target rather than a wildcard.
- the check now
- Inspect the live packet-filter ruleset immediately after enabling any new rule. A rule reading as an any-to-any match where something specific was intended is an immediate rollback signal.
An API that accepts a request and reports success is not the same thing as an API that understood the request, and this is a clean demonstration of the gap. Two field names, chosen by reasonable guesswork rather than checked against the API’s actual schema, were close enough to look right and different enough to be dropped without comment. What got stored was a rule missing the two fields that mattered most, the destination and the port.
A translation rule with no destination and no port does not fail. It matches everything. The firewall did exactly what it was told, which was the problem. It was told the wrong thing, cheerfully, with a success response at every step of the chain. Six minutes later, on 20260906, the public services were back, and the pair’s own configuration sync had helpfully propagated the mistake to the second firewall in the meantime.
The practical fix generalises to any infrastructure-as-code workflow against an API that is lenient about unknown fields, which is most of them. Never treat a saved response as proof that the fields were understood. Create the object disabled or in a dry-run state, read it back through the same API, and compare it field by field against what was intended before it goes live. Then check the effect rather than the object: read the running configuration the device actually enforces, and look for the specific target you meant to write.
The cost of the read-back is a few seconds. The cost of skipping it, here, was every public service redirected to one machine, with nothing anywhere reporting an error.
Source: node0 lessons v0.1, lesson 4.14. Sanitized: checklist v0.1, 20260921; addresses, target host name; voice pass 20260921. Part of oznog.com/node0.