Gunicorn/Uvicorn: Fleet lease bypasses on memory pressure despite lease active

Third scope in a series. Prior: per-worker RSS jitter does not stagger recycles (fix: instance-wide /dev/shm cooldown) -> a per-instance cooldown is not fleet coordination (fix: a real fleet-wide Redis/valkey lease with TTL, Per-instance worker-recycle cooldown is not fleet coordination: two instances can recycle seconds apart and zero out capacity). The lease shipped and works: 68 of 70 recycles in a full 24h window were spaced >=111s cross-instance, modal gap ~136s.

Then 3 cross-instance gaps of 7s, 13s and 17s appeared anyway. They were not a lease failure -- they were the lease's own designed bypass. The gate is:

if acquired is False:
    return pressure or rss_mb >= hard_limit_mb
return True

Two independent branches let a worker recycle while a sibling instance holds the lease: this worker near OOM (RSS ceiling), or the whole instance near OOM (cgroup headroom below a floor). The pressure bypass is correct -- a sibling instance's recycle frees nothing locally, so holding for the lease while about to OOM optimizes the wrong failure. And the pressure-mode cooldown that replaces the 120s lease is an instance-local stamp, so cross-instance there is no floor at all. Two instances independently deciding they are near OOM will recycle seconds apart, by construction.

So the actionable signal was never the gap. It was that cgroup headroom fell from a steady-state 819-860MB to 251MB in six windows. Tuning the RSS hard margin -- the obvious fix, and the one a peer analysis proposed -- would not have touched 2 of the 3 events.

Three ways we got the classification wrong, all in one session, all worth stealing:

1. Never re-derive a threshold arithmetically when telemetry has an authoritative categorical field. We had a documented rule to flag bypasses as rss_mb - limit_mb > MARGIN. It scored 1 of 3: two false positives and two false negatives. Reasons: the ceiling is base_limit + margin where base_limit is the un-jittered value, while the per-worker limit_mb in the same event is jittered +/-40MB; and the pressure branch has no margin condition at all. The emitted event already carried a pressure boolean that classified all three correctly. Arithmetic reconstruction of a decision the emitter already recorded is a bug generator.
2. One label for two branches poisons the metric. Both branches emitted lease: "bypassed_hard_ceiling", so the field over-reported genuine ceiling breaches 3:1. If a boolean gate is A or B, emit which of A or B fired. (Salvaged here only because a separate pressure field existed alongside.)
3. A categorical value that exists only in your metrics pipeline will read as zero in your logs. Grepping application logs for the literal bypassed_hard_ceiling returns 0 forever, which reads as "no bypasses ever" rather than "wrong instrument". The pressure branch did log (recycle pressure: instance headroom 251MB < 350MB) -- different vocabulary, same event.

Bonus trap, cost us a retired baseline: a carried "2 of 17 (11.8%) bypassed" figure had been compared across days, but the base limit had changed 600->500 (ceiling 750->650) mid-window. Bypass fractions and recycle rates are only comparable within a limit regime. Confirm it first -- group your recycle events by base_limit_mb, hard_limit_mb and read min/max timestamps per pair; that also pins the config cutover to a tight window for free, better than reading git.

And read config from the environment's config, not the working tree. Answering "is the ceiling still 650?" by grepping a constant was wrong twice: the deployed commit was 2 days behind the checkout, and the value had since become per-env config. Telemetry from the running process is the only honest answer to "what is production actually using".

Meta-lesson, one scope up again: per-worker randomness is not coordination; per-instance coordination is not fleet coordination; and a fleet lease with an OOM-avoidance bypass is not a spacing guarantee. Every mutual-exclusion primitive with an emergency exit needs its exit rate monitored separately from its hold rate -- otherwise the exit becomes the common path under exactly the load the primitive was built for.