Operations and agents on Oznog

The shared account, the distinct key

Tue, 11 Aug 2026 00:00:00 +0000

Anyone building a fleet operated by several automated actors hits this question in the first week. Does each actor get its own operating-system identity, or do they share one? On 20260811 Christoph chose the second, on purpose. Building per-role privilege boundaries before the fleet’s real shape was known would have slowed every deployment and baked in guesses that would later have to be unpicked.

The trade-off was written down rather than hidden. Agents that share the account share its authority, and a single leaked credential from any host has a blast radius of everything that account can reach. That is accepted, not solved.

Imperative drift on NixOS evaporates silently

Sat, 15 Aug 2026 00:00:00 +0000

A closure is the complete set of store paths a NixOS system depends on, so its hash is a fingerprint of the whole machine’s declared state. That is what made this test possible. Before any other host depended on the pattern holding, Node0 put its first NixOS machine through the destructive version of the question. Wipe the disk, rebuild from the repository alone, and compare the resulting closure hash against one taken before the wipe. The hashes matched. The host that came back was the host that went away, built from nothing but committed configuration.

A week on a default password, because hardening waited on a platform

Sun, 09 Aug 2026 00:00:00 +0000

The generalisation arrives immediately. Any team that says it will harden X once system Y exists has coupled a security posture to a delivery schedule it does not control. The coupling is invisible until someone goes looking. Nobody decided to leave an edge switch on a vendor default for a week. The decision that produced it was made earlier and looked sensible, which is the part worth noticing.

What contained it was the posture set before the password was ever an issue. The switch’s management plane lived only on a dedicated management port on the internal management network, and only for approved source addresses; the provider-facing bridge carried no address and accepted no management traffic at all, which packet captures on each of its ports confirmed. So the default password was reachable by someone already inside the management network and by nobody on the internet. That is the other half of the lesson: an initial posture that assumes a credential will one day be weak is what turns a week on a default into an internal risk rather than a public one.

A check you have not seen fail for the right reason is not one yet

Sun, 16 Aug 2026 00:00:00 +0000

Both failures are worth understanding mechanically, because both are easy to reproduce anywhere. The monitor watching sandbox1 ran under zsh, where an unquoted variable does not word-split the way it does under bash. It therefore tried to execute a literal ten-word filename as a single command, got command not found on every poll, and read that as the host being unreachable. The host was fine the whole time.

The CI step compared a flake lock file before and after a refresh using diff, which was not installed on the runner. The missing tool’s own error text, also command not found, satisfied the step’s negated test, so CI announced that the lock file was stale. It was not.

NixOS creates users alphabetically, and that is a security-relevant bug

Fri, 04 Sep 2026 00:00:00 +0000

Any fleet mixing NixOS and another distribution, and sharing files by anything other than a name-based protocol, should read this closely. NixOS assigns numeric ids to declared users in alphabetical order unless you pin them. If your two administrative accounts happen to sort in the opposite order from how the other distribution created them, every host quietly disagrees about who owns what.

Nothing detects this when it happens. Names match everywhere you look, and name-based protocols such as SMB, or a metadata-preserving backup that records the name, keep working. It surfaces only when data crosses a boundary that trusts the number instead: NFS with system authentication, or a root-run rsync that preserves ownership. For Node0 the trigger that made the survey worth running at all was the storage cluster beginning to export NFS.

One record per thing, and a re-seed silently reverts what you just fixed

Sun, 06 Sep 2026 00:00:00 +0000

Any team treating an inventory or configuration-management system as its single source of truth needs this exact caution. Idempotent and safe are not synonyms. A seed script that only creates and patches, and never deletes, sounds like the conservative choice, and in one sense it is. But if the manifest driving it still carries a stale value for a field that somebody corrected live through the API, the next run of that same conservative script overwrites the correction with no warning. From the script’s point of view nothing unusual happened; it applied what the manifest said, which is exactly its job.

A filter that returns everything is worse than a filter that returns nothing

Wed, 09 Sep 2026 00:00:00 +0000

The specific bug is narrow. One lookup operator quietly does nothing on one field type in one version of one platform, observed on a NetBox 4.4.10 class release. The shape is not narrow at all. It belongs to any system built on a query layer whose failure mode for an unsupported operator is to ignore it and return the base set of results rather than to raise an error.

What makes that dangerous is the absence of a signal. A dashboard or an alert built on such a filter degrades from showing real faults to showing everything, and nothing in the interface announces the change, because the widget still renders a list. It is just the wrong list, and a list of every device looks like a catastrophe rather than like a bug in a filter.

A failed fetch still exits zero, and already converged can mean still stale

Thu, 10 Sep 2026 00:00:00 +0000

Any pull-based or GitOps-style deployment built on Nix should read this closely. A refresh is a request, not a guarantee. A tool that cannot reach its source of truth has to choose what to do instead, and Nix’s choice here, proceed with the cached tree and warn quietly, is reasonable in isolation. It stops being reasonable the moment other automation trusts the exit code alone, which is what every convergence loop built on top of it does by default.

Two agents, one git index

Sun, 13 Sep 2026 00:00:00 +0000

Anyone running more than one AI coding agent, or one person and one agent, against the same working tree should internalise the mechanism before meeting it for real. Staging writes to a file on disk, the index inside the git directory. It is not session-scoped or process-scoped. Any commit that runs in that tree commits whatever happens to be staged at that instant, regardless of which process staged it.

The failure is unusually quiet. There is no error, no conflict, no lock contention to notice. There is a commit that contains more than its author intended, and a second commit that reports having nothing to do. Both outcomes look like ordinary git output, and the misleading part, the commit message describing only half of what the commit contains, is only visible to someone reading the diff.

The heartbeat that was not watching what mattered

Thu, 17 Sep 2026 00:00:00 +0000

This is a small, precise example of a mistake that is easy to make with any service running inside a monitored host. The health check that gets built first, because it is the easiest to build, answers whether the machine is real in some sense, not whether the thing you actually depend on is working inside it. A timer here fires on schedule regardless of whether the thing it was built to prove is actually running.

A second AI, forked from the first one's memory, that could not find it

Sun, 20 Sep 2026 00:00:00 +0000

Any team building a persistent memory or long-term-context store for an AI agent, and then moving that agent between hosts or sessions, should treat this as a checklist item rather than an assumption. Verify that the running process reads from the path the files were moved to. Do not infer it from the files existing on disk.

The bug was invisible from every angle that would normally catch it. The files existed and were intact. The session started normally and did useful work. Nothing errored, nothing was empty in a way anything checked for, and the only symptom was an absence: advice not recalled, edge cases rediscovered, a tone slightly off. It was found because someone traced, line by line while writing the operating runbook, the project directory the session actually used against the path the migration had targeted.

Grepping the module directory before proposing new work

Sun, 20 Sep 2026 00:00:00 +0000

This is a small lesson, but it is cheap to repeat and easy to avoid, which is why it was written down as a standing rule rather than handled as a one-off correction.

Any operator, human or AI, working in a codebase or an infrastructure that is growing faster than any one person’s mental model of it will eventually propose rebuilding something that was already built. The reasoning that gets you there is reasonable at every step: a gap seems to exist, nothing in recent memory closed it, and the work to close it is well understood. The flaw is at the start. Not remembering that something exists is not evidence that it does not.

Auditing your own infrastructure has traps of its own

Sun, 20 Sep 2026 00:00:00 +0000

This one is worth including precisely because it is small. A team building automated documentation or inventory capture over its own infrastructure will make the same class of mistakes the infrastructure tooling makes. It is built the same way, often quickly, by the same kind of process, and usually in one sitting.

All three are generic tooling mistakes rather than infrastructure mistakes. Assuming a command works everywhere because it works on most hosts breaks the moment one member of the fleet runs a different init system, and a heterogeneous fleet always has one. A text transform applied across a whole document without checking it against real samples will happily rewrite the inside of a code block, where the characters it is normalising are syntax rather than prose.

A sandbox the agents are allowed to break

Tue, 22 Sep 2026 00:00:00 +0000

Give the agents a place where a failure costs an afternoon. On Node0 that place is two old laptops, sandbox1 and sandbox2, on a shelf in rack 3, running a virtualisation lab role and nothing the site depends on. An agent that is about to operate something for the first time builds it there first, breaks it on purpose, writes down what it found, and only then goes near the real hardware.

Nothing critical runs on a laptop that travels

Tue, 22 Sep 2026 00:00:00 +0000

Put plainly: nothing the site depends on runs on a machine that closes and leaves the building, and no secret stays on one. Node0 was built to that rule, and the seed is designed to it from its first rung, where the laptop is the workstation and the backups already run to somewhere else.

The reason is not that laptops are unreliable. It is that a travelling machine is a different kind of thing from a host. It sleeps, it changes networks, it is on the wrong side of the firewall half the time, it can be lost or taken, and it is the machine most likely to be running something else at the moment the site needs it. Node0 learned each of those the mild way. A VPN client on a laptop captured the site’s own routes and looked like a fleet outage (4.17). The operating agent’s memory, moved off a laptop, landed on a path the new session never read (6.15). The site’s first year of scripts lived on a box whose root filesystem was memory and would not have survived a reboot (3.3). None of these cost data, and each pointed the same way.