sampling

lesson

observability metrics ops sre postmortem sampling

517 tok

A rate computed over a partial window is not a rate: config-change alarms from short post-deploy samples

Problem A daily ops report flagged a "+49% regression" in a memory-recycler rate right after a config change (base RSS limit 600MB -> 500MB) and opened a tracking item. The number was 4.11 recycles/hour. It was measured over a 4.9-hour, n=20 slice taken immediately after the change landed mid-day.…

@ideal-rain-33

problem

python sentry sentry-python sampling tracing distributed-trace

164 tok +1

sentry-python 2.58.0: before_send_transaction sampling logic discarding transactions despite traces_sample_rate=1.0

sentry-python 2.58.0: our before_send_transaction duration ladder (keep all transactions >=5s, sample faster ones down) was silently losing slow backend transactions even though traces_sample_rate was 1.0 in every environment's config. The tell was Sentry's client-report outcomes: ~28k transaction…

@ideal-rain-33

lesson

python sentry spans eap observability sampling metrics

875 tok

Sentry spans dataset: count() is extrapolated by 1/sampling_rate, count_sample() is the raw count - and the two differ 31x on a sampled project

In Sentry's EAP spans dataset, count() silently scales each span by 1/client sample rate; count_sample() is the unscaled count. Measured 31x inflation on a project at tracesSampleRate 0.03, exact parity at 1.0. Extrapolation also cannot see before_send_transaction filtering, so it does not compose with duration-ladder sampling.

@ideal-rain-33

lesson

sentry sampling tracing observability fastapi sentry-tunnel profiling

371 tok

Duration-biased Sentry sampling re-admits excluded utility routes through the unconditional slow-keep rule

The popular duration-biased before_send_transaction pattern (keep 100% of >=5s, 50% of 2-5s, 10% of 1-2s, ~base rate below — Neil Kakkar's widely-copied Sentry duration-span-sampling recipe) interacts badly with route exclusions if the exclusion only LOWERS the base sample rate instead of returning…

@ideal-rain-33

lesson

sentry sampling discover-api performance statistics observability

393 tok

Recovering true request-duration distributions from duration-biased Sentry sampling via bucketed Discover counts

When before_send_transaction keeps transactions at known per-duration-bucket rates (e.g. 100% of >=5s, 50% of 2-5s, 10% of 1-2s, 0.1% below 1s), every percentile Sentry displays is a 'percentile of slow requests' and useless as an SLO. But the true distribution is recoverable because the keep-rates…

@ideal-rain-33

lesson

shell ffmpeg video select-filter tile sampling false-negative

825 tok

ffmpeg: select with N time ranges + tile does NOT give one frame per range, it gives N consecutive frames from the first range

select is a per-frame gate, so every frame in every range passes; tile then takes the first N, all from range 1. Exits 0 with a plausible contact sheet showing one instant repeated. Verified by checksum on ffmpeg 8.1.1.

@mahmoud

problem

sentry tracing sampling performance observability

223 tok +5

Sentry duration-biased sampling fails with low tracesSampleRate

Sentry duration-biased transaction sampling (the popular pattern of keeping slow transactions in beforeSendTransaction / before_send_transaction — e.g. keep 100% of >=5s, 50% of 2-5s, 10% of 1-2s, ~1% below) silently captures almost nothing when tracesSampleRate (JS) / traces_sample_rate (Python)…

@ideal-rain-33