Patch the fleet in waves, and mean it

Rollout · June 9, 2026 · 6 min read

Almost nobody argues against staged rollout. Almost nobody executes one. What usually happens is that a change goes to a handful of machines somebody happens to have access to, sits there for a day or so while nothing in particular is observed, and is then released to everything else on the grounds that the first group “looked fine”. The vocabulary of staging is present. The mechanism is not.

The mechanism is worth recovering, because staging is expensive. It multiplies the calendar time of every rollout, it forces the estate into a state where two versions coexist, and coexistence has its own failure modes. If you are paying that, you should get something back.

What the first batch is for

The first batch is not a smaller version of the rollout. It is an experiment, and it has to be designed like one.

Two pressures pull against each other. The batch must be small enough that its total failure is affordable — that if every host in it becomes unusable, the organisation absorbs it without a crisis. And it must be diverse enough that its success is informative. Those pressures point in opposite directions, which is why the batch that gets chosen is so often the wrong one: a set of identical, freshly built, lightly loaded hosts belonging to the team running the rollout. It is small and it is safe and it tells you nothing, because the failure you are looking for lives in the six-year-old machine with the vendor agent, the unusual filesystem layout, and the driver nobody can find a source for.

A useful first batch contains at least one instance of every awkward thing you know about. It is not the easy hosts. It is a sampler of the estate’s variety, kept small by taking one or two of each kind rather than by taking only the simple kind.

This is also the argument against percentage-based batching taken literally. Ten per cent of the fleet is a quantity, not a sample. If it is selected by hostname order, and hostname order correlates with build date — as it usually does — the first batch is a single cohort, unrepresentative in a direction nobody noticed.

What a soak is for

Soak time is the interval between finishing one batch and starting the next. Its purpose is to let failures become visible before the population exposed to them grows.

That gives it a precise sizing rule, which is almost never applied: the soak should be at least as long as the latency of the slowest failure mode you care about. Some failures declare themselves in seconds — the service does not start, the host does not come back. A soak of minutes catches those. Others are patient. A change that breaks a nightly batch job is invisible until the night. One that breaks a monthly reconciliation is invisible for weeks. One that only manifests under the load pattern of a Monday morning cannot be seen on a Thursday afternoon at any duration.

So the question to ask before setting a soak is not “how long is reasonable?” but “what is the longest-latency failure I am trying to catch, and does this soak span it?” If the answer is that the failure you fear takes a month to appear, you are not going to soak for a month, and that is fine — but then you should stop pretending the soak covers it and find another control, such as being able to reverse the change a month later.

The second rule is blunter. A soak during which nothing is being watched is not a test, it is a delay. Before the batch goes out, name the signal: which metric, which log, whose report. If the answer is “we would hear about it”, the soak is decorative and its calendar time is being spent on nothing.

Stopping is the whole point

Here is the part that quietly invalidates most staged rollouts. A stage boundary is a place where you can stop. If you cannot stop, the boundary is a pause, and a pause has none of the value that justified the cost.

Being able to stop means two separate capabilities. The first is that you can decline to proceed — that batch four is not already in flight through an automation that nobody remembered to gate. The second is that you can reverse what batches one to three received. Only the first is usually true.

Reversal is where the assumptions break. Uninstalling a package is not a rollback; it reverses files, not state. The upgrade may have migrated a configuration format, altered an on-disk schema, or been accompanied by a peer component that moved forward to match. Removing the new version leaves the old binary looking at data it does not understand, which is frequently worse than the bug you were escaping. Firmware raises the stakes further: a device with a single flash bank has no reverse gear at all, and a device with two banks has one only if the fallback path has been exercised recently enough to trust.

The honest test of a rollback plan is whether it has been executed — not reviewed, executed — on a host in the same state as the ones you would need it for. Rollback rehearsed on a clean machine in the afternoon and rollback attempted on a half-applied machine at three in the morning are different procedures with different success rates, and only one of them has been tested.

When reversal genuinely is not available, say so explicitly and plan differently. Roll forward instead: keep a tested route to a newer version that fixes whatever the current one broke, and make sure that route can be travelled faster than the problem spreads. That is a legitimate strategy. Pretending you have a rollback you have never run is not.

Decide the abort criteria first

Whatever the criteria are, write them down before batch one, because the pressure not to stop increases with every batch that succeeds. Three batches in, with a deadline behind you and a fourth already scheduled, the same ambiguous signal that would have halted the rollout on day one gets reclassified as noise. This is not a failure of character; it is what sunk cost does to everybody. The only reliable countermeasure is to have committed to the threshold while it was still cheap to be cautious.

Criteria should be things you can observe during the soak you actually chose. “No increase in error rate” is checkable. “No user impact” is not, unless you have said how you would know.

The arithmetic underneath

All of this reduces to a small piece of scheduling. The fleet divides into batches; each batch costs a patch run plus a soak; the batches have to fit into windows somebody else controls. Those four numbers determine whether the plan you have described is a week of work or an impossibility, and the difference is visible before you start.

The staged rollout planner does that arithmetic, including the two cases people get wrong: the short final batch, and the plan where a single batch plus its soak will never fit inside a single window.