A VCR-backed e2e test (LLM pipeline, vcrpy match on method+uri only) recorded green but failed on replay with responses consumed against the wrong requests (per-URL FIFO shift, cascading dspy ChatAdapter->JSONAdapter fallbacks that burned further interactions).
Root cause: the test harness loaded real dev secrets only when recording (if record_mode != 'none': load_dev_secrets()). Code gated on if api_key: (a FRED factcheck + LLM consistency judge) ran at record time but was skipped at replay, making one fewer LLM call per generation unit. With vcrpy matching on method+uri, all same-endpoint calls form one positional queue, so a single skipped call shifts every subsequent response.
Fixes:
- Give the test env a NON-EMPTY placeholder for every secret that gates a code path (
fred_api_key_secret: 'test-placeholder-key'); VCR's filter_query_parameters/filter_headers strip the key from matching so the placeholder still matches recorded interactions. Rule of thumb: record/replay must execute the identical call sequence; anyif api_key:branch must resolve the same way in both modes. - Kill wall-clock in replay-reachable gates: a staleness check compared
datetime.date.today()against recorded API observation dates, so cassettes rot as real time passes the max-age window, silently flipping gate outcomes (and thus LLM call counts) weeks after recording. Thread a pinned now_date from the pipeline's logical date instead.
Diagnosis technique that cracked it: monkeypatch vcr.cassette.Cassette.play_response in conftest to log each play with a fingerprint of request vs response schema fields (for dspy, regex \[\[ ## (\w+) ## \]\]); the first request whose response fields don't match its own request fields marks the exact skipped call. Also note: vcrpy logs 'Appending request' at INFO during cassette LOAD (Cassette._load calls append), which looks alarmingly like recording during a replay run but is not.