The PG autoscaler that silently did nothing, because two rules overlapped
- date
- 20260911
- what happened
- Ceph's automatic placement-group sizing tool had been returning a blank result for some time, which read as nothing needs adjusting but actually meant it had refused to act on any pool in the cluster. One internal management pool had been assigned a placement rule spanning every hardware class, which overlapped the class-scoped rules every other pool used, and Ceph silently refuses to make sizing decisions across pools with overlapping rule scopes.
- what it cost
- An unmeasured period with the cluster's automatic capacity sizing doing nothing at all while appearing merely quiet. Once corrected, six pools needed resizing and the resulting rebalance moved roughly 27 TiB over seven hours, work a functioning autoscaler would have spread out gradually.
- what changed
- The overlapping pool was moved onto a class-scoped rule matching the others, unblocking the autoscaler cluster-wide. Six pools were then resized, though live figures later showed some ended up larger than the plan's stated target, itself flagged as a discrepancy to reconcile rather than silently accepted.
- the check now
- A non-empty autoscale-status table is now the health signal for the autoscaler itself, not just its recommendations. A blank result is investigated as a tool failure, never read as nothing to do.
An automated tuning system that stops reporting recommendations looks exactly like a system that has decided everything is already correct, and those two states require opposite responses. One needs nothing; the other needs someone to notice the tool has stopped working at all.
Here, a capacity-management pool inside a distributed storage cluster had at some point been assigned a placement policy broader than everyone else’s, which put it in conflict with a safety rule that refuses to make sizing recommendations across any set of pools whose placement scopes overlap. The storage system enforced that rule silently and, from its own point of view, correctly. It declined to act rather than acting on inconsistent information, and gave no indication anywhere that this was what was happening rather than the benign explanation that everything was already sized well.
Narrowing that one pool’s placement scope back in line with the rest of the cluster immediately unblocked sizing recommendations for every pool at once, which is itself a useful diagnostic signal. If enabling the autoscaler on one pool produces a sudden flood of recommendations across many unrelated pools, an overlapping rule elsewhere is a likely explanation. The broader lesson travels past storage entirely. Any automation that can go silent for either of two very different reasons, agreement or being blocked, needs its own health signal separate from the absence of output.
Source: node0 lessons v0.1, lesson 3.18. Sanitized: checklist v0.1, 20260921; cluster and pool identifiers; voice pass 20260921. Part of oznog.com/node0.