Symptom
A service ran two replicas behind one platform log stream. A memory-recycler audit read the recycle timestamps and found three inter-event gaps under the 120s cooldown (min 35s, four distinct PIDs inside ~3 minutes) and escalated a possible cooldown breach.
There was no breach. Per-replica the intervals were 129s and 138s, both clear.
Cause
The platform CLI (render logs, v2.20.0) merges all replicas into one chronological stream and exposes no instance label. Worse, the escape hatches don't work: -o is rejected as an unknown flag, and --json still emits text with no labels[] array, so there is nothing to demultiplex on.
With N replicas, a merged stream compresses apparent inter-event gaps by roughly a factor of N. Any per-entity rate, interval, cooldown, or debounce conclusion drawn from it is wrong by construction, not merely unlabeled — and it fails in the alarming direction, manufacturing violations of exactly the timing invariants you are auditing.
Diagnostic
The tell is cheap: count distinct PIDs in the suspicious window and compare to workers-per-replica. Four PIDs in three minutes against a 3-worker-per-replica config means at least two replicas are represented, so the gaps are cross-entity. That gets you to "probably not a breach" but not to a number.
Procedure
- Get identity from a source that carries it. Here a structured product event (PostHog
worker_recycled) carriedrender_instanceandpidper record, which demultiplexed cleanly:
06:25:53 replica-w4cmv pid 2048 # \
06:26:28 replica-kslp4 pid 1790 # | merged stream shows 35s, 94s, 44s
06:28:02 replica-w4cmv pid 1912 # |
06:28:46 replica-kslp4 pid 1970 # /
-> per replica: w4cmv 129s, kslp4 138s- Route by question, not by convenience. Aggregate volume and presence/absence questions are fine on a merged stream ("did
WORKER TIMEOUToccur at all"). Anything involving time between events for one entity must come from an instance-labeled source. - Write the routing rule into the runbook. Two agents independently burned a round trip on this in one session because the constraint lives in the CLI's behaviour, not anywhere a reader would look.
- If you emit the telemetry yourself, put the replica identity in the event. It costs one field and it is the difference between an answerable and an unanswerable question.
Generalization
Before differencing timestamps, ask what the population is. A stream that interleaves independent entities does not support per-entity interval statistics, and the corruption is invisible: every timestamp is real, every gap is arithmetically correct, and the conclusion is still false. Same family as computing a minimum over a metric whose extreme values only one code branch can produce.