Follow-on to the memory-based gunicorn recycling lesson whose fix was an instance-wide cooldown stamp in /dev/shm (per-worker RSS jitter does not stagger recycles). That fix works exactly as designed and is worth keeping. It also has a ceiling that only appears once you run more than one instance.
/dev/shm is per-container. A cooldown stamped there is shared by the forked workers of ONE instance and is invisible to every sibling. The invariant you actually bought is 'at most one recycle per instance per cooldown', not 'at most one recycle in the fleet per cooldown'.
Observed on a 2-instance x 3-worker deployment: instance A recycled at 12:47:58.046Z (RSS 655MB) and instance B at 12:48:04.677Z (RSS 637MB), 6.6s apart.
Verification detail that matters and will otherwise send you chasing the wrong bug: the platform log stream MERGES instances. A naive scan shows sub-cooldown pairs and looks like the cooldown itself is broken. Split by the instance label (-o json, then labels[] name=="instance") before judging. Every same-instance pair was >120s apart — the cooldown was intact and the defect was one scope up.
Why instances synchronize is the same mechanism as the original per-worker trap, one level up: instances behind a round-robin balancer receive near-identical traffic, so their memory creeps at near-identical rates and they cross the threshold together. Per-instance randomness would help as little as per-worker randomness did.
Blast radius: a 14-second window where in-flight SSR requests exceeded a 10s upstream timeout, producing user-visible 504s. Two of six workers leaving is survivable in isolation; it is not survivable when a crawler already has the pool near saturation (our p50 upstream fetch during crawl windows was 3.0s against a 10s budget). Note the interaction — the recycler and the crawler were both ours, and neither was individually misbehaving.
Fixes, cheapest last: (a) a real fleet-wide lease (Redis or a Postgres advisory lock with TTL) so at most one instance recycles at a time; (b) derive a phase offset from a stable instance identity and require now % period to land in your slot — no shared state needed; (c) at minimum, refuse to drain when healthy local capacity is already below a floor.
Meta-lesson, restated one scope up from the original: per-worker randomness is not coordination, and per-instance coordination is not fleet coordination. Every time you scale out a tier, re-ask which scope your mutual-exclusion primitive actually covers.