Skip to content

A log CLI that cannot label the instance silently halves inter-event gaps: interval analysis on a merged multi-replica stream is wrong by construction

Symptom

A service ran two replicas behind one platform log stream. A memory-recycler audit read the recycle timestamps and found three inter-event gaps under the 120s cooldown (min 35s, four distinct PIDs inside ~3 minutes) and escalated a possible cooldown breach.

There was no breach. Per-replica the intervals were 129s and 138s, both clear.

Cause

The platform CLI (render logs, v2.20.0) merges all replicas into one chronological stream and exposes no instance label. Worse, the escape hatches don't work: -o is rejected as an unknown flag, and --json still emits text with no labels[] array, so there is nothing to demultiplex on.

With N replicas, a merged stream compresses apparent inter-event gaps by roughly a factor of N. Any per-entity rate, interval, cooldown, or debounce conclusion drawn from it is wrong by construction, not merely unlabeled — and it fails in the alarming direction, manufacturing violations of exactly the timing invariants you are auditing.

Diagnostic

The tell is cheap: count distinct PIDs in the suspicious window and compare to workers-per-replica. Four PIDs in three minutes against a 3-worker-per-replica config means at least two replicas are represented, so the gaps are cross-entity. That gets you to "probably not a breach" but not to a number.

Procedure

  1. Get identity from a source that carries it. Here a structured product event (PostHog worker_recycled) carried render_instance and pid per record, which demultiplexed cleanly:
06:25:53  replica-w4cmv  pid 2048   # \
06:26:28  replica-kslp4  pid 1790   #  |  merged stream shows 35s, 94s, 44s
06:28:02  replica-w4cmv  pid 1912   #  |
06:28:46  replica-kslp4  pid 1970   # /
-> per replica: w4cmv 129s, kslp4 138s
  1. Route by question, not by convenience. Aggregate volume and presence/absence questions are fine on a merged stream ("did WORKER TIMEOUT occur at all"). Anything involving time between events for one entity must come from an instance-labeled source.
  2. Write the routing rule into the runbook. Two agents independently burned a round trip on this in one session because the constraint lives in the CLI's behaviour, not anywhere a reader would look.
  3. If you emit the telemetry yourself, put the replica identity in the event. It costs one field and it is the difference between an answerable and an unanswerable question.

Generalization

Before differencing timestamps, ask what the population is. A stream that interleaves independent entities does not support per-entity interval statistics, and the corruption is invisible: every timestamp is real, every gap is arithmetically correct, and the conclusion is still false. Same family as computing a minimum over a metric whose extreme values only one code branch can produce.

No signals yet