Lazy per-worker sentence-transformers model load masquerades as a memory leak under RSS-based gunicorn worker recycling

Symptom: RSS-recycle rate on a FastAPI/gunicorn API jumps (0 -> 14/24h) with no deploy; max observed worker RSS climbs (539 -> 670MB); request-count recycling (--max-requests 2000) stops firing because workers hit the RSS limit first. Looks exactly like a fresh leak.

Discriminators that settle leak-vs-spike from logs alone (no new instrumentation needed if the recycler emits a periodic per-pid heartbeat of rss/reqs/uptime):

1. Long-lived workers (8-10h, 2200+ requests) hold FLAT or declining RSS -> not request- or time-proportional growth, so not a leak.
2. RSS trajectories are step functions: quantized jumps between adjacent samples (+55-76MB and +120-195MB in our case), not slopes.
3. Requests-at-death spread is huge (15 to 2189 in the same fleet): a worker can die within 15 requests of boot. A true per-request leak gives a narrow requests-at-death band.
4. Deaths cluster in active-usage hours, tracking workload bursts round-robined across workers (2-3 workers jump within minutes of each other).

Mechanics (gunicorn --preload, 3 workers, sentence-transformers all-MiniLM-L6-v2, 384-dim, ~87MB fp32):

• torch/transformers are imported at module level -> pre-fork in the master -> boot RSS ~483MB but COW-shared. Fine.
• The MODEL loads lazily per worker on first embed (SentenceTransformer(...) behind a global) -> +55-76MB private pages per worker, not shared, fanned across workers as a burst distributes requests.
• First encode() adds activation buffers + OpenMP thread arenas (torch default nthreads, uncapped) -> +120-195MB in one step. glibc arenas do not return freed memory to the OS, so RSS parks above the new floor; the next spike crosses the jittered soft limit and the recycler (correctly) kills the worker.
• A single quiet day after a deploy reads as "pressure resolved"; the next active day reads as "pressure returned". Neither is real: steady state is spike-driven and workload-correlated.

Fixes, cheapest first:

1. Accept it: if recycles are graceful (SIGTERM-to-self, drain, respawn behind siblings) and the hard ceiling/cgroup pressure paths never fire, ~10-20 recycles/day is harmless. Recalibrate alerts to zero-tolerance only for hard-ceiling bypasses, pressure-mode recycles, and OOM kills.
2. Load the model pre-fork (module import or app factory under --preload): weights become COW-shared across all workers, eliminating both the per-worker jump and N-1 private copies.
3. torch.set_num_threads(1) (or OMP_NUM_THREADS=1) to stop first-encode arena bloat; MiniLM inference on short texts doesn't benefit from intra-op parallelism on 1-CPU instances anyway.

Meta: before hunting, check whether your "clean baseline" reading was actually a low-traffic day. Bucket recycle events by hour-of-day; workload-correlated clustering rules out monotonic leaks in minutes.