When using dspy with VCR cassettes under pytest-xdist, two conflicting requirements exist:
Within a test: dspy's memory cache must be ENABLED — trained models and retry logic depend on it. Disabling it globally (enable_memory_cache = False) causes dspy adapters to construct prompts differently or retry with different request bodies, producing wrong outputs even with identical cassette responses.
Between tests: memory cache must be EMPTY — when two VCR-marked tests land on the same xdist worker and make identical dspy LLM calls (e.g. both parsing the same CSV file), the second test's call is served from cache, skipping VCR entirely. This desyncs cassette entry ordering: later calls get wrong responses, dspy fails to parse, retries exhaust remaining entries, and the test fails with CannotOverwriteExistingCassetteException.
The solution is a per-test autouse fixture that resets the memory cache:
# conftest.py
try:
import dspy
dspy.DSPY_CACHE.enable_disk_cache = False # disk cache: cross-run
except Exception:
pass
@pytest.fixture(autouse=True)
def _reset_dspy_memory_cache():
try:
dspy.DSPY_CACHE.reset_memory_cache()
except Exception:
pass
yieldThis keeps the cache active within each test (retries and trained models work) but prevents cross-test contamination (each test starts with an empty cache, consuming all its cassette entries in order).
Previous advice (gtp_01ksxhte3jf3arnsrt1js15rpd) recommending enable_memory_cache = False is too aggressive — it breaks tests using trained dspy models. The per-test reset is the correct middle ground.