Grading an observability fix: three ways silence lies, and what to measure instead

You change error reporting — add a fingerprint, downgrade a severity, suppress a noise class, guard a crash — ship it, and check back later. The issue is quiet. Fix accepted.

That inference is invalid in three situations that are not rare, and all three appeared in a single triage session.

1. The capture site is unreachable, so the fix never executes

A classifier was written to route a specific class of fetch failure to warning level under a stable fingerprint with a companion tag. It was unit-tested, wired at the correct call site, reviewed, and deployed. Three days later it had executed zero times.

Every caller caught its own exception first and never re-raised:

try:
    response = httpx.get(...); response.raise_for_status()
except Exception as e:
    log(...); capture_exception(e); return []      # swallowed before the new wiring

That identical block appeared in 42 of 45 sibling modules. The exception never reached the instrumented layer. Reading the changed file could not reveal this; the defect lived in the callers.

Grade by the output, not the code path. If a fix is supposed to start emitting a tag, the tag is the test:

has:my_new_tag     14d -> 0    90d -> 0    all-env 90d -> 0

Zero everywhere, across cycles that demonstrably ran, means it never ran. Note this only works if you added a tag alongside the fingerprint — Sentry does not index fingerprints and there is no fingerprint: search token, so a fingerprint alone leaves you with no handle in the UI, in saved searches, or in alert rules. Add the companion tag at the same time you add the fingerprint, or you cannot grade your own change.

"Not yet testable" has an expiry. The moment you can show a cycle ran on a fix-carrying release, the verdict is FAILED, not pending. For scheduled work, prove the cycle ran by naming a sibling issue on the same schedule that fired in the same window — the job running is a separate fact from your fix working, and conflating them is how a broken fix survives a review.

2. The alert has a recovery latch, so silence is guaranteed

A health check alerted on unhealthy data sources like this:

if not extras.get('health_alerted_at'):
    capture_exception(SourceUnhealthyError(...))
    extras['health_alerted_at'] = now      # cleared only on recovery

A source that stays broken emits exactly one event, ever. Fourteen sources failed simultaneously, produced fourteen events in one second, and then nothing — while the underlying outage continued indefinitely. Quietness here is not evidence of repair; it is a structural certainty, and it would look identical if the situation got ten times worse.

Before writing "stays at zero" as acceptance for anything, grep the capture site for an already-alerted flag, a dedup key, a cooldown, or a once-per-incident guard. If one exists, the acceptance signal has to be out of band: a database column advancing, rows appearing downstream, a counter — something the latch does not gate. Write that into the tracking note explicitly, because the next person to look will otherwise read silence the obvious way.

This is the mirror image of a rule worth stating alongside it: when the producer is removed (a deprecated detector, a deleted code path), silence is also meaningless, but in the opposite direction — those issues are bookkeeping and should be closed rather than watched.

3. The fix intentionally leaves a residual, so it can never go to zero

A client-side filter suppressed stale-asset errors on their first occurrence (a reload is imminent) and deliberately kept reporting two other cases: errors arriving through a path where no reload is coming, and errors that persisted after a reload already happened. The second class is not leakage — it is the fix's designed output, meaning "we reloaded and the asset was genuinely gone."

A rate criterion cannot separate those from failure. A composition criterion can:

In a 100-event sample, every event on a post-fix release with mechanism X carries tag Y.

Build it by partitioning the sample on release with git merge-base --is-ancestor <fix> <event_release>, then comparing tag distributions. The measured result:

Zero leakage on post-fix releases. That is checkable in one call, it fails loudly, and — the part that matters — it would have caught the original bug on day one. The filter had been dead for six weeks because it tested mechanism.type === 'sveltekit' while the SDK emitted auto.function.sveltekit.handle_error; a rate criterion never noticed, because the volume looked plausible either way.

Pair it with a rate clause that excludes deploy days. A deploy legitimately produces real version skew, and counting those days as regressions trains you to ignore the metric.

The common shape

Silence is a single bit, and every one of these failures produces that bit for a reason unrelated to the fix. The replacements are all the same move: measure a distribution over a sample — which tags appear, on which releases, from which clients — instead of a count over time. It is one API call in each case, and unlike waiting a week, it gives an answer the same day.