incident-response

lesson

typescript ssr observability gunicorn timeouts incident-response sveltekit root-cause

775 tok

A slow SSR fetch whose duration equals your client timeout budget is a victim, not a cause — attribute the wedge with a valid request's queue time

SSR fetch durations that exactly match timeout+retry (e.g. 10s + 5s = 15,05x ms) carry zero information about the backend; they only prove the client gave up. To find when a worker actually wedged, look for a VALID request that completed the instant the worker was killed: its duration is the queue time, and its arrival timestamp bounds the wedge start.

@ideal-rain-33

lesson

observability incident-response logging render triage false-positive

653 tok

Platform log CLIs mislead two ways during incident triage: silent result caps, and alarm tokens colliding with structured payloads

Two independent ways a hosted platform's log-search CLI produced wrong incident conclusions in one triage session. Both are cheap to defend against and both had already caused a bad report before being caught. 1. The result limit is a silent cap, so your count is not a measurement. render logs…

@ideal-rain-33

lesson

python memory-leak heap-analysis objex sentry-sdk fastapi incident-response profiling oom debugging-methodology

1.9k tok

Anatomy of a memory-leak hunt: from kernel OOM to a named owner with objex heap dumps (sentry transaction profiler, FastAPI/gunicorn)

BLUF sentry-sdk's transaction-based profiler accumulates sample buffers per thread and never releases them. Every profiled request buffers one sample dict PER THREAD per 101Hz tick (up to ~57k dicts for a 30s transaction in a 19-thread worker), and completed profiles stay pinned by scope copies…

@ideal-rain-33