A secret rotation that silently did nothing
- date
- 20260907
- what happened
- The NixOS module that builds NUT's ups.conf renders that file into the world-readable Nix store by default, so Node0's module instead renders it at runtime from a template with placeholders. When the underlying SNMPv3 secret was rotated, the rendered file did not update, because systemd's restartTriggers fire only when the template changes, and the template's text was identical before and after.
- what it cost
- A rotation that appeared to succeed, with every systemd unit reporting active, but had not taken effect. It was found only by noticing that the UPS monitoring driver was returning no data at all.
- what changed
- The secret's unit now declares an explicit restartUnits dependency, so a secret-only change triggers the render step, and consumers are ordered PartOf that render unit.
- the check now
- Verifying a render takes two checks, not one: the rendered runtime file must have its placeholders replaced, and the store copy of the template must still show the placeholders unchanged, proving the secret never touched the store.
This is a narrow trap with a wide blast radius, for anyone combining NixOS secret management with a module that renders configuration text at runtime.
The setup is ordinary and correct. Some NixOS options write their rendered configuration file straight into the Nix store, which is world-readable on the host, so a secret placed there is readable by any local user. The usual workaround is to keep a template in the store with placeholders where the secrets go, and have a small unit substitute the real values into a runtime file at service start.
The failure is in how the restart is triggered. A systemd restartTrigger compares the rendering instructions, not their output. Rotate the secret and leave the template byte-identical, which is the normal case, and nothing changes from systemd’s point of view, so nothing restarts and nothing re- renders. The service keeps running against the stale value. It does not fail. Every unit reports active. The only symptom here was that the NUT driver returned no data at all, and that was noticed rather than alerted on.
Two corrections follow. First, declare the trigger against the secret itself, not only the template, so a secret-only change fires the render step, and order the consumers PartOf that step so they follow it. Second, verify every rotation at both ends, because each end catches a different mistake. The live rendered file must show the new value with no placeholders left. The store’s copy of the template must still show only placeholders. The first proves the rotation took. The second proves you did not just publish the secret to every user on the machine.
Source: node0 lessons v0.1, lesson 5.9. Sanitized: checklist v0.1, 20260921; module path; voice pass 20260921. Part of oznog.com/node0.