Storage and backups on Oznog

Never delete in the same command as an unverified copy

Thu, 23 Jul 2026 00:00:00 +0000

An AI agent working through a large file-reorganisation project destroyed three of Christoph’s voicemail recordings in a single command. The command was meant to move files to a new location and clean up the source, but the move used a remote path with quote characters that a modern scp treats literally rather than expanding through a shell. The copy silently failed while looking like it might have worked. The delete that followed in the same command ran anyway, because nothing had checked whether the copy actually landed.

The library that is not a copy: an external-library rule for Immich

Fri, 24 Jul 2026 00:00:00 +0000

Self-hosted media tools that promise not to duplicate your library often do exactly what they say. They hold pointers, not copies. Immich was installed to manage roughly 135,000 photos and videos (counted at setup, 20260724), and to keep storage sane it was pointed at existing folders rather than given a fresh copy of everything. That is the efficient design and also the trap. From inside the application, an externally referenced photo and an internally stored one look identical. There is no visual cue that says “if you delete the file behind this thumbnail, the photo is gone everywhere, including from here”.

A RAM root: a year of scripts that never survived a reboot

Sun, 26 Jul 2026 00:00:00 +0000

An appliance-style NAS operating system such as Unraid runs its entire root filesystem in RAM by design, which is normally invisible. The home directory behaves like a real one, survives SSH sessions, and df reports plenty of free space. Nothing about using it signals that a reboot erases it completely. A year’s worth of document-import scripts and an API credential had quietly piled up there. They were discovered only because someone happened to go looking, and they survived only because the box had gone eighteen days without a restart.

The scratch directory setting that the helper program ignored

Sun, 26 Jul 2026 00:00:00 +0000

A container platform with a size-capped storage backend fails in a particularly confusing way. The application filling the disk can be entirely correctly configured, and the disk still fills, because a program it invokes underneath does not honour that configuration. Paperless, a document manager, had its own scratch-directory setting pointed at the right place. What filled the disk was the OCR tool Paperless calls out to, which uses the operating system’s generic temp-directory convention and ignored Paperless’s setting entirely.

The append-only backup server's orphan-pack loop

Mon, 27 Jul 2026 00:00:00 +0000

Backup systems built to resist deletion, an append-only mode so that a compromised client cannot erase history, create a failure mode that looks nothing like what it is. A network hiccup during an upload left data blocks on the server with no index entry pointing at them, orphaned rather than committed. Because the server refuses to let a client overwrite anything, the retry could not simply replace the failed upload. It kept recomputing the identical blocks and getting refused every time, which looked exactly like an overloaded server rejecting work under pressure. A day went into chasing that theory before the retry succeeded at an even higher load, disproving it.

The comment character hiding at the front of a folder name

Thu, 06 Aug 2026 00:00:00 +0000

A batch file-deletion tool’s list format borrowed a very old Unix convention. A line beginning with # is a comment, not data. That convention collided with a personal filing system that prefixes its top-level folders with # for visual grouping in a file browser. Because the path list was built relative to each subtree rather than from a common root, many entries legitimately started with that character, and the tool silently treated roughly a quarter of the intended deletions as comments, then reported complete success.

Trash is a choice, not a default, and the two tools disagree

Thu, 06 Aug 2026 00:00:00 +0000

Two tools that both call themselves “delete” can mean opposite things. A higher-level sync tool trashes files by default, giving a 30-day recovery window. The cloud provider’s own raw API call named files.delete permanently destroys the file with no such window, a distinction stated plainly in the provider’s documentation but easy to miss when rewriting a script for speed.

The rewrite here was a reasonable engineering choice: moving from a general-purpose sync tool to a direct API call for bulk deletion by file ID. It just silently traded away the safety property the earlier version had, and nothing about the code’s shape signalled the change. Because a completely separate discipline was already in place, verifying every file by checksum against another copy before deleting anything, no data was actually lost. But the operator had been told the deletions were reversible when they were not, which is its own kind of failure even without a loss to point at.

The NAS boot flash failed, and the backup built to survive it

Sat, 08 Aug 2026 00:00:00 +0000

An appliance NAS keeps its entire personality, network configuration, container definitions, user accounts and host keys, on a small internal USB flash drive separate from the data disks. That flash drive failed. The only reason this was recoverable at all was luck. An old backup happened to still exist from months earlier, written by a script nobody had thought carefully about.

That old script had a design flaw invisible until the day it mattered. It wrote its backup onto the very box it was protecting, so if that box’s boot media dies, the backup dies with it. The replacement design is a short checklist for backing up any single-purpose appliance’s own identity: encrypt it, because configuration backups tend to contain credentials by necessity, and put every copy somewhere that survives the box itself being dead, not just somewhere convenient.

What an offsite drive protects, and the guard that let it overflow anyway

Tue, 18 Aug 2026 00:00:00 +0000

A capacity check that correctly identifies a coming failure is only useful if something actually stops when it fires. Here the check logged the right number, sent the right alert, and then execution fell through into the very operation it had just warned against. The code path that logs a warning and the code path that halts execution were two different things, and only one of them existed. The immediate fix was one missing line.

A hand-edited config that hides its own drift for weeks

Tue, 18 Aug 2026 00:00:00 +0000

Many long-running daemons read their configuration once at startup and never again. That is an entirely reasonable design choice, and it becomes a trap the moment someone edits the file directly instead of going through the application’s own change mechanism. Here a hand edit correctly landed on disk, and the process kept running on its stale in-memory copy with no indication anywhere that the two had diverged.

The dangerous part is the timing. The edit does not simply fail to take effect; it becomes a change staged to activate at some unknown future moment, whichever routine event happens to restart the process next, an operating system update, a crash, a planned reboot. That turns an ordinary maintenance action into an unplanned configuration change days or weeks later, with no correlation an operator would think to check.

The exclude that used to be true

Tue, 18 Aug 2026 00:00:00 +0000

A backup exclude list is not really a list of paths. It is a list of claims about what those paths currently contain, and those claims silently go stale the moment a path is reused for something else. An exclude written correctly for an abandoned profile folder would have become, unnoticed, an exclude for an entire active working profile the day that folder was renamed onto the same location. The backup job would have kept reporting success the whole time, because from the tool’s point of view nothing had gone wrong. It was faithfully honouring an instruction that had simply stopped meaning what it once meant.

A backup nobody has restored is a belief, and skipped is not passed

Fri, 21 Aug 2026 00:00:00 +0000

Backup verification tends to stop one step short of the thing anyone actually cares about. It is easy to confirm a snapshot was taken, and not much harder to confirm the backup’s internal bookkeeping is structurally sound. Neither proves that a real file, fetched over the network, decrypted and reassembled, comes back out intact. This system was built to close that last gap: an actual small-scale restore every week, compared against what was recorded, rotating through the whole fleet’s repositories over time rather than testing the same one repeatedly.

Everything moves off the old NAS: the migration and its USB boot saga

Thu, 03 Sep 2026 00:00:00 +0000

A brand-new server, meant to become the central hub for an entire home infrastructure, could not reliably boot its own operating system, and the reason was almost invisible. The small USB flash drive holding the OS worked perfectly in the machine’s firmware boot menu and then vanished the instant the Linux kernel took over. That happened only on the faster USB 3.0 ports, and only some of the time.

Diagnosing this meant watching console output directly, since the machine’s own network access was down and remote log-reading was not an option. It also meant hand-navigating a UEFI shell to boot while working out what was happening. The eventual answer was not a firmware setting or a bad cable. It was a known-class incompatibility between that drive and that USB controller generation under Linux specifically, a combination that had no reason to be tested before the hardware was already committed to a migration deadline.

The recursion rule: a backup of a backup is worse than no backup

Sat, 05 Sep 2026 00:00:00 +0000

A multi-tier backup design, where several systems back each other up in different directions to spread risk across independent failure domains, has one obvious way to quietly fail. A loop, where system A’s backup of system B includes data that originated from A in the first place, gives the appearance of redundancy while providing none. It consumes real capacity and produces real confidence, and neither is worth anything.

The rule adopted here is simple to state and easy to violate by accident as a system grows, so it was written into the tool that generates backup configurations rather than trusted to human memory or to a document nobody rereads. Every plan the generator produces is checked against the rule before it is allowed to run, and a plan that would violate it, even through an indirect path introduced by some later change, fails to validate and the nightly run refuses to proceed.

Rebalancing a ZFS pool: the math nobody runs before doing it anyway

Sat, 05 Sep 2026 00:00:00 +0000

Storage systems that stripe data across multiple physical groups of disks tend to attract an intuitive but wrong mental model: add an empty group, and surely writes will preferentially fill it until things even out. The reality, measured directly here rather than assumed, is a mild statistical bias, not a redirect. An empty group picks up only a modest premium over its full neighbours, nowhere near enough to correct a real imbalance within any reasonable amount of data movement.

The bucket that was declared purged and was not

Sat, 05 Sep 2026 00:00:00 +0000

Migrations and redesigns leave behind old artefacts, and the natural response is to note that this will clean itself up and move on to the actual new work. That note is a prediction, not an observation, and predictions about background processes are exactly the kind of thing that silently fails to come true with nobody watching.

Here, an old object-storage bucket from a replaced design was assumed to be draining itself through a purge operation that had never been configured to run at all. No lifecycle rule existed for it. The bucket sat there for at least six days, continuing to accrue storage cost, discovered only because a later, unrelated piece of work looked at storage usage again and noticed the number had not moved.

Editing the config by hand took out both nightly backups at once

Wed, 09 Sep 2026 00:00:00 +0000

A system built with real safety mechanisms can still produce a bad night. The mechanism did not fail. Someone worked around the process that keeps the mechanism informed. Two backup plans, writing to two entirely separate destinations, looked independent from the outside. Both actually depended on one shared step earlier in the night: a single atomic snapshot of all the data, taken once and read by both, specifically so that neither backup captures data mid-write.

The PG autoscaler that silently did nothing, because two rules overlapped

Fri, 11 Sep 2026 00:00:00 +0000

An automated tuning system that stops reporting recommendations looks exactly like a system that has decided everything is already correct, and those two states require opposite responses. One needs nothing; the other needs someone to notice the tool has stopped working at all.

Here, a capacity-management pool inside a distributed storage cluster had at some point been assigned a placement policy broader than everyone else’s, which put it in conflict with a safety rule that refuses to make sizing recommendations across any set of pools whose placement scopes overlap. The storage system enforced that rule silently and, from its own point of view, correctly. It declined to act rather than acting on inconsistent information, and gave no indication anywhere that this was what was happening rather than the benign explanation that everything was already sized well.

One unique password, and a backup the verifier could not see

Thu, 17 Sep 2026 00:00:00 +0000

Security intuition and operational reality pulled in opposite directions here, and operational reality won for a specific, well-reasoned cause. Giving a new machine its own unique backup-encryption password is, in isolation, the more cautious choice: no single compromised secret affects more than one host. But the fleet’s automated proof that backups are actually sound, a nightly and weekly sweep that opens every repository and checks it, was built around a shared password with a short, explicitly maintained list of exceptions.

The setting named keep less was the one that removed the protection

Sun, 26 Jul 2026 00:00:00 +0000

Cloud object storage services often expose version retention as a small handful of named presets, and the names can be actively misleading about what security property each one provides. The first mistake here was a cost problem, not a security one. The default keeps every version forever, silently billing for space a backup tool believed it had already reclaimed by pruning.

The fix for that, choosing the option that sounded like the tightened version of the same idea, quietly introduced a much worse problem. Once a file version is hidden, it becomes permanently and irreversibly gone after one day. Hiding requires only a write permission, not a delete permission, an important and easy-to-miss detail in how these systems distinguish the two. That defeats the point of keeping hidden versions at all, which is to survive a scenario where the credential doing the writing has been compromised and is being used to destroy data. With a one-day window, an attacker barely needs to act quickly.

The second storage tier nobody was writing to

Sun, 20 Sep 2026 00:00:00 +0000

Some infrastructure gaps are loud: something breaks, an alert fires, someone investigates. This one was silent in the most complete way possible. A whole layer of a backup design existed, was correctly built, had a working restore path proven against it, and simply had no job configured to ever put anything into it. Nothing about that state produces an error. The storage server reports healthy. The datastore reports available capacity. The one backup that ever ran, a manual test weeks earlier, succeeded and sat there looking exactly like evidence the system worked, when it was the entire extent of what the system had ever done.