The scratch directory setting that the helper program ignored
- date
- 20260726 to 20260801
- what happened
- Unraid keeps Docker storage in a fixed-size image file of 50 GB. A container's writable layer filled it, not its images: Paperless's own scratch directory was configured correctly, but Paperless shells out to a separate OCR tool that ignores that setting and writes to the standard /tmp, which lives inside the same fixed-size image.
- what it cost
- Two outages of the Docker platform, at 71% and then 72% full, each stopping every container on the host and each requiring a live investigation, because reading the environment variable alone looked correct.
- what changed
- The scratch directory was moved onto its own ZFS dataset and mapped into the container's /tmp, taking it out of the fixed-size image, out of the nightly offsite backup set and out of snapshots, deliberately, since it is pure scratch. Moving Unraid off a fixed-size Docker image entirely was identified and scheduled rather than done immediately.
- the check now
- Measure inside the container (docker exec <container> du -sh /tmp) before declaring a scratch-directory fix complete, because a helper program the application calls can write outside its setting entirely; use docker system df and docker ps -as to separate image bloat from writable-layer bloat.
A container platform with a size-capped storage backend fails in a particularly confusing way. The application filling the disk can be entirely correctly configured, and the disk still fills, because a program it invokes underneath does not honour that configuration. Paperless, a document manager, had its own scratch-directory setting pointed at the right place. What filled the disk was the OCR tool Paperless calls out to, which uses the operating system’s generic temp-directory convention and ignored Paperless’s setting entirely.
The first fix addressed only Paperless’s own setting, and the outage recurred days later from the same underlying cause. The real fix was to stop trusting any single application’s configuration and instead measure disk usage directly inside the running container, which revealed a four-gigabyte orphaned directory from the OCR tool that no configuration file mentioned.
The permanent solution moved the scratch space onto its own real filesystem location outside the platform’s size cap altogether, so the failure mode cannot recur regardless of what any individual tool does with its temp files. Two habits carry over to any Docker host with a fixed-size backing store: separate image bloat from writable-layer bloat before theorising, and treat an application’s temp-directory setting as covering only the code that application actually runs itself.
Source: node0 lessons v0.1, lesson 3.4. Sanitized: checklist v0.1, 20260921; dataset paths, container names; voice pass 20260921. Part of oznog.com/node0.