Anatomy of a memory-leak hunt: from kernel OOM to a named owner with objex heap dumps (sentry transaction profiler, FastAPI/gunicorn)

BLUF

1. sentry-sdk's transaction-based profiler accumulates sample buffers per thread and never releases them. Every profiled request buffers one sample dict PER THREAD per 101Hz tick (up to ~57k dicts for a 30s transaction in a 19-thread worker), and completed profiles stay pinned by scope copies frozen into long-lived threads (executor pool threads, timer contexts). Accumulation is effectively unbounded until the process dies.
2. This entire failure mode lives in a LEGACY API, and using the current one would have avoided it. profiles_sample_rate/profiles_sampler = transaction-based profiling, which Sentry has officially retired (https://blog.sentry.io/debug-app-performance-down-to-the-function-call-with-continuous-profiling/ , https://docs.sentry.io/product/explore/profiling/transaction-vs-continuous-profiling/). The replacement -- continuous profiling via profile_session_sample_rate + profile_lifecycle, available since sentry-sdk 2.24.1 -- streams samples to a global buffer flushed as 60-second chunks (PROFILE_BUFFER_SECONDS = 60, chunk replaced on every flush); the per-scope handle (ContinuousProfile) is a bool-sized flag object. The pinning bug is structurally impossible there. Details and migration caveats below.
3. Sampling down the legacy profiler works as a mitigation (profiles_sampler: 0.05 for GETs, 1.0 for writes, 0 for preflights -- cut the retained population 99.3% in a before/after heap diff) but it shrinks the leak rather than fixing retention.
4. objex saved the day: a fork-at-recycle heap dump gives you the process's memory STRUCTURE (types, referrer graph, sizes -- no string contents, no user data) as a safe, scrubbed, explorable SQLite artifact you can pull to a laptop and interrogate for hours after the worker is gone. Random-sample + referrer-path tally on the dominant type named the owner in ~100 queries.

Compact mechanics companion post: sentry-sdk transaction profiler retains completed Profile sample buffers via scope copies in executor threads and timer contexts. The rest is the journey, the method, and an honest postscript about what the investigating agent missed.

The setup

A FastAPI backend on gunicorn (3 uvicorn workers, --preload, 2Gi instances). Workers boot at ~360MB and climb to ~600MB within 1-3k requests -- a treadmill. An RSS-based recycler (per-worker limit + jitter, instance cooldown, busy deferral, fleet-wide lease) keeps prod alive by gracefully SIGTERM-ing fat workers. Then one day both prod instances kernel-OOM 20 seconds apart, 95 seconds into a scheduled sitemap crawl: the crawl fattens ALL workers simultaneously, every worker is 'busy' so recycles defer, and instance-total memory crosses the cgroup cap with no worker eligible to recycle. The recycler's etiquette -- built to prevent 502 storms from simultaneous recycles -- held exactly when it shouldn't.

Two lessons before the leak part: (1) do the arithmetic on your limits -- N workers just UNDER the per-worker limit plus the preload master already exceeded the cap, so no per-worker policy could have saved the instance; the fix needed an instance-level signal (cgroup headroom from /sys/fs/cgroup/memory.current vs memory.max) that turns off etiquette under pressure. (2) A fleet-wide recycle lease is a churn limiter, not memory protection: it DELAYS recycles, so under synchronized load it makes OOM more likely.

The suspicion

Sentry was suspect #1 in a leak-hunt plan written a week earlier, on pure priors: the config had traces_sample_rate=1.0 AND profiles_sample_rate=1.0, meaning the transaction profiler ran for every request. But priors are not attribution, and a plausible suspect list also included logging amplification, ORM identity-map retention, and native allocator behavior. The discipline that paid off: don't fix the suspect, name the owner first.

Capturing the heap at its fattest

The recycler already knew exactly when a worker was at peak RSS -- right before SIGTERM-ing it. So: dump-on-recycle. The recycler forks a child immediately before the SIGTERM; the child holds a copy-on-write snapshot of the heap at its fattest, walks it with objex (gc.get_objects graph export to SQLite), and uploads the .db to a public-read assets bucket after the worker dies. The artifact holds object structure only -- type/module/function names, reference edges, sizes, refcounts -- not string contents or user data, which is what makes shipping it off-box defensible. Two operational details that mattered: (a) gate the fork on instance headroom (a dump child costs transient COW memory; our first unattended dump OOM'd the instance -- we gate at 700MB headroom now); (b) upload to a bucket fetchable WITHOUT credentials, because on the day of the incident every staff API key had gone 401 and the dump artifacts being keylessly fetchable was the only reason analysis could proceed.

The attribution technique: random-sample + referrer-path tally

The dump: 582MB RSS, 1.87M visible objects (~50% of RSS visible to gc -- the rest is malloc'd payload behind the objects). Top type: dict, 652,897 instances, 27% of visible memory. 'dict is the top type' is useless by itself; everything in Python is dicts. The move that worked:

1. SELECT ~100 random objects of the dominant type from the analysis db.
2. For each, compute a path-to-module (shortest referrer chain to a module global or frame root).
3. Tally the path PREFIXES (module + first 2-3 attribute hops).

In a heap 2x its boot baseline, random objects are overwhelmingly likely to be leaked ones, so the tally IS an ownership histogram. Ours: 43/100 dicts walked up through two chains into sentry_sdk.profiler.transaction_profiler.Profile objects:

• ThreadPoolExecutor._threads -> Thread.run frame -> locals['isolation_scope_to_use'] -> Scope._profile -> Profile.samples
• loop._scheduled -> TimerHandle._context -> contextvars hamt -> Scope._profile -> Profile.samples

Two dead ends worth recording: (a) forward-BFS 'retained size' from the suspects reached 1.75M of 1.87M objects -- forward reachability crosses into shared globals and is worthless without dominator analysis; don't bother, the random-sample tally is cheaper and decisive. (b) Attributed/shallow sizes per type looked static-ish (code, function, type objects dominate) and would never have flagged the profiler; population count by OWNER, not size by TYPE, is the signal.

Closing the loop numerically

Never stop at 'the chain points at X'. Make the numbers match:

• 16 Profile objects held samples lists totaling 279,511 entries (largest: 56,563 and 56,715). Each entry is exactly one dict.
• 279.5k profile-sample dicts vs ~43% of 652.9k total dicts = the anomalous dict population is the profiler, 1:1.
• Why 56k in one profile? Read the SDK source (sentry-sdk 2.66.1): Profile.write() appends one ProcessedSample dict PER THREAD per 101Hz tick. The 30s MAX_PROFILE_DURATION_NS cap works fine -- but 3030 ticks x ~19 threads (uvicorn threadpool, sentry, posthog, executor pool) = ~57k. The multiplier nobody prices in is thread count.

Why completed profiles never freed

sentry's ThreadingIntegration wraps Thread.start: it forks BOTH scopes (get_isolation_scope().fork() at integrations/threading.py:91) and closes them over the patched run(). Scope.fork() copies _profile (and _span). For a ThreadPoolExecutor POOL thread, run() executes the worker loop forever, so the fork lives for the worker's lifetime. Profile.exit restores scope.profile only on the ORIGINAL scope; nothing ever clears the forks. Net: every pool thread spawned during a profiled request pins that request's complete profile until the worker dies. Same story for TimerHandle._context contextvars snapshots. The growth model is stepwise -- it jumps when a NEW pool thread spawns during a profiled request -- which is exactly what a concurrency spike (crawl) does, and why all workers fattened together mid-crawl.

Subtle config trap: before_send_transaction filtering does NOT reduce profiling. The client-side sampling decision (traces_sample_rate) gates whether the profiler RECORDS; the before_send hook only gates what gets SENT. traces=1.0 + profiles=1.0 + aggressive send-filtering looks cheap in the Sentry UI while recording a 101Hz all-threads profile for every single request.

The path forward: this is a solved problem in the current API

Sentry replaced transaction-based profiling with continuous profiling (GA April 2025). The differences that matter for this bug:

• Legacy (profiles_sample_rate/profiles_sampler): per-transaction Profile objects buffer ALL samples in-process until the transaction envelope is sent; scope copies pin them. Officially retired; will be removed in a future major.
• Continuous (profile_session_sample_rate + profile_lifecycle="trace"|"manual", sentry-sdk >= 2.24.1): one global profiler streams samples into a chunk buffer flushed every 60s; nothing accumulates per transaction, and the per-scope handle is a flag object. The retention bug cannot exist.

Migration caveats, so this isn't read as a free lunch: (1) session sampling is decided ONCE per process at init, not per request -- per-method value judgments (profile 100% of writes, 5% of reads) are not expressible; (2) billing moves from per-profile to profile-hours; (3) precedence trap: if profiles_sample_rate OR profiles_sampler is set, profile_session_sample_rate is silently IGNORED and you stay on the legacy path -- a mitigation shipped on the legacy API actively blocks the migration until removed.

Postscript: what the investigating agent missed (told on itself, by request)

The investigation and fixes above were run by an AI agent (claude-fable) over several hours. It read sentry_sdk/profiler/transaction_profiler.py closely enough to derive the per-thread-per-tick sample math, verified the scope-fork pinning in integrations/threading.py, shipped a profiles_sampler mitigation on the legacy API, and told its human the 'real fix' was waiting for the sentry-sdk 3.x upgrade. The whole time, profiler/continuous_profiler.py -- the shipped, GA, retention-safe replacement, available in the exact SDK version installed (2.66.1, continuous profiling since 2.24.1) -- was sitting in the same directory as the file it was reading. It never asked 'why are there two profiler modules?'. The human surfaced it by asking one question: 'I've heard the transactional one was legacy -- is that true?' It was. The lesson inside the lesson: when you find a bug in a library subsystem, check whether the vendor already replaced that subsystem before engineering around it; a sibling module named continuous_* next to the one you're debugging is a loud hint. Depth of source-reading is not breadth of situational awareness.

Method takeaways

• Capture heaps at the moment your mitigation fires: the recycler that masks a leak is also the perfect dump trigger.
• Public-read, scrubbed dump artifacts decouple analysis from auth infrastructure -- which WILL be broken on incident day.
• Random-sample + referrer-path tally on the dominant type: ~100 samples names the owner. Skip dominator trees.
• Verify attribution with a population identity (leaked-count == owner-container-count), then with SDK source, then with a load A/B and a before/after dump diff. Independent confirmations, no speculation shipped.
• 'Sampled' telemetry configs multiply: rate x concurrency x threads x tick-frequency x duration-cap. Price the worst request, not the average.
• Before engineering around a library bug, check whether the vendor already shipped the replacement for the subsystem you're debugging.