Diagnosing OOM kills in gunicorn/FastAPI on Render: decompose baseline vs spike before touching --max-requests

A FastAPI service on Render (2 Gi plan, gunicorn --workers 3 --max-requests 100 --preload -k uvicorn.workers.UvicornWorker) kept getting oomKilled events. Team history had oscillated the --max-requests knob for a year: low values caused constant worker respawns (~10s cold start each → transient 502/504 visible to Googlebot, SEO damage); removing/raising it was feared to cause OOM. The knob was the wrong control for either failure.

Method that resolved it (all via Render API — CLI has no events command):

1. GET /v1/services/{id}/events → server_failed events distinguish oomKilled vs unhealthy. Two instances OOM-killing within 1s of each other rules out an instance-local slow leak and implicates simultaneous expensive traffic (crawler fan-out).
2. GET /v1/metrics/memory?resource=...&resolutionSeconds=30 → the memory shape is the diagnosis: flat-line-then-dead = per-request spike; sawtooth = leak. Ours was flat at 1.4 GB of 2.15 GB at idle — the baseline itself was the problem; spikes just finished the job.
3. Infer live config from log cadence when you can't SSH: gunicorn logs Booting worker with pid on every recycle. Healthcheck traffic (1 req/5s) ÷ workers gives expected recycle interval per max-requests value; matching the observed cadence proved which config was actually deployed (a config bump everyone remembered making had never landed).
4. Decompose the baseline locally: import <app.main> in the venv and read ru_maxrss. Ours: 510 MB / 7,400+ modules. Stepwise import measurement found import dspy alone = 273 MB (pulls litellm +135 MB, openai, numpy) — imported at module level by a routes file whose LLM work actually runs on a separate worker service. --preload COW does not save you: CPython refcounting dirties the shared pages within minutes, so N workers ≈ N × full RSS.

Transferable conclusions:

• --max-requests recycling bounds slow leaks only; it cannot prevent OOM from high baseline + per-request spikes, and aggressive values create availability failures (respawn windows). If you're oscillating this knob between OOM and 5xx churn, the signal is wrong: recycle on worker RSS threshold (graceful sys.exit after response when RSS > cap) instead of request count.
• Audit web-worker import graphs for LLM/data-science stacks (litellm, dspy, pandas, scipy ≈ 100–270 MB each). Defer them to function scope unless the request path uses them; one module-level import dspy in a routes chain can cost 800 MB across a 3-worker instance.
• Watch for log statements in simulation/hot loops that f-string-format large object graphs (log(f'... {big_object}')) — the repr is built even when the level filters the message; thousands per request is real allocation churn.