A week on a default password, because hardening waited on a platform
- date
- 20260809 to 20260816
- what happened
- The network hardening of the WAN edge switch finished on 20260809, but rotating its shared default administrator had been routed through a secrets platform whose own build then slipped. A production, internet-facing switch sat on a vendor default for about a week, not by oversight but because a security task had inherited a dependency on something that was not being built yet.
- what it cost
- Roughly a week on a vendor default on the device that faces the WAN, reachable only from inside the internal management network. No compromise is known, and none was possible from the internet, but the risk window was real and was accepted rather than noticed.
- what changed
- Christoph decoupled routine security work from the secrets platform entirely on 20260816: keys for routine access, a password manager for custody, the vendor default kept only as a verified, key-gated break-glass path. A written standard and a CI gate followed on 20260905.
- the check now
- The one-sentence test is now a CI test. If rotating a password requires editing the repository, the design is wrong. The break-glass default is only valid while password authentication over SSH is structurally refused for any account holding a key, and that coupling is written down per device.
The generalisation arrives immediately. Any team that says it will harden X once system Y exists has coupled a security posture to a delivery schedule it does not control. The coupling is invisible until someone goes looking. Nobody decided to leave an edge switch on a vendor default for a week. The decision that produced it was made earlier and looked sensible, which is the part worth noticing.
What contained it was the posture set before the password was ever an issue. The switch’s management plane lived only on a dedicated management port on the internal management network, and only for approved source addresses; the provider-facing bridge carried no address and accepted no management traffic at all, which packet captures on each of its ports confirmed. So the default password was reachable by someone already inside the management network and by nobody on the internet. That is the other half of the lesson: an initial posture that assumes a credential will one day be weak is what turns a week on a default into an internal risk rather than a public one.
Christoph’s fix was not to rush the platform. It was to remove the dependency. Every device and every operator gets a distinct key for routine access. Custody lives in a password manager that exists today rather than in a plan for a future vault. The vendor default survives only as a break-glass path whose safety is verified rather than assumed, which in practice means the device must refuse password authentication over SSH for any account that holds a key.
OpenBao did eventually arrive, as a three-member raft cluster initialised on 20260912, a year of scope later than the incident that had been waiting on it. By then it inherited these habits instead of replacing them. The abandoned single-node placeholder that had briefly existed earlier was retired rather than migrated, and starting clean on the final topology cost nothing, because nothing production had ever been stored in it. A dependency you can remove is cheaper than a dependency you can only wait out.
Source: node0 lessons v0.1, lesson 6.3. Sanitized: checklist v0.1, 20260921; device model, password-manager name, paths; voice pass 20260921; exposure scope verified against the device runbook 20260922. Part of oznog.com/node0.