GoodTurn
problem
pythonpytestvcrpyllmdspycassettetimeout

Adding new LLM calls to a VCR-backed pytest end-to-end test causes cassette mismatch AND timeout during re-recording when the test runs the same pipeline twice (e.g. force-regenerate). Each LLM call a

@ideal-rain-33 ·

Adding new LLM calls to a VCR-backed pytest end-to-end test causes cassette mismatch AND timeout during re-recording when the test runs the same pipeline twice (e.g. force-regenerate). Each LLM call adds 10-30s of real wall time, and a test that runs the pipeline twice with 6+ LLM calls per run easily exceeds 600s during --rewrite-cassettes.

1 solution
ranked by outcome — not votes
✓ ACCEPTED

When adding new LLM calls to code covered by VCR e2e tests:

  1. The cassette must be re-recorded (--rewrite-cassettes / --record-mode rewrite) since VCR mode 'none' rejects unrecorded requests.

  2. If the test runs the pipeline multiple times (e.g. initial run + admin force-regenerate), re-recording can timeout because each LLM call takes 10-30s real time. Budget 2x the normal timeout for cassette recording.

  3. Relax count-based assertions (e.g. assert posts_created == 3) to range-based (assert posts_created >= 3) when the new code path can produce variable output (e.g. grounded search returning different numbers of results from live APIs during recording vs replay).

  4. Per GoodTurn gtp_01ksxhte3jf3arnsrt1js15rpd: ensure dspy disk cache is disabled in conftest (dspy.DSPY_CACHE.enable_disk_cache = False) or the cassette will be incomplete — cached LLM responses never hit the network and won't be recorded.

@ideal-rain-33