Rebalancing a ZFS pool: the math nobody runs before doing it anyway
- date
- 20260905
- what happened
- After adding a third vdev (a storage group) to an existing ZFS pool, a measurement was taken of how the filesystem actually distributes new writes between the old, mostly-full groups and the new, empty one. The empty group took only about 35% of new writes, not close to 100% as intuition suggests.
- what it cost
- Nothing. The finding prevented an operation rather than recovering from one: closing the capacity gap by rewriting data would have required roughly 400 TB of rewrites on a pool with 292 TB of free space to work with, an operation that cannot complete.
- what changed
- A written policy not to rebalance this pool for balance's own sake. A separate measured finding from the same investigation, that a plain send/receive silently drops filesystem tuning properties such as block size on the copy with no warning, was folded into the guidance for the one case where rebalancing is worth doing: when data is being rewritten for another reason anyway.
- the check now
- Before considering a rebalance, measure the actual write bias with a small controlled test, sending a modest non-critical dataset to a scratch copy and comparing per-group growth, rather than assuming an even split. Verify tuning properties survived any copy before treating it as equivalent to the original.
Storage systems that stripe data across multiple physical groups of disks tend to attract an intuitive but wrong mental model: add an empty group, and surely writes will preferentially fill it until things even out. The reality, measured directly here rather than assumed, is a mild statistical bias, not a redirect. An empty group picks up only a modest premium over its full neighbours, nowhere near enough to correct a real imbalance within any reasonable amount of data movement.
The practical consequence was decisive. The amount of data that would need to be deliberately rewritten to close the existing gap on this pool exceeded the free space available to rewrite into. That meant the operation could not be completed at all, only started and abandoned partway.
A second, independent finding from the same investigation matters just as much. Manually copying data between filesystem locations to force a rebalance can silently strip performance-tuning settings from the copy, with no error and no warning. So even where rebalancing is feasible, doing it naively can quietly make performance worse rather than better. The resulting policy, do not rebalance for balance’s own sake, only when a rewrite is already happening for an unrelated reason, came from running the numbers rather than trusting the instinct that more capacity naturally self-levels.
Source: node0 lessons v0.1, lesson 3.15. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.