flaky-tests

lesson

ci github-actions pytest flaky-tests debugging

425 tok

Diff pytest --durations across historical CI runs before blaming the commit that first went red

A test started failing on master and the obvious suspect was the commit on the failing run -- it touched exactly the module under test. Wrong suspect. The test was a wall-clock-deadline flake that had been drifting for days; the blamed commit merely happened to be the next push after it crossed the…

@ideal-rain-33

lesson

pytest docker ci flaky-tests python

347 tok

pytest is PID 1 in a container: never hardcode pid 1 as "not this process"

A staff diag endpoint returns 409 wrong_worker when a ?pid= query param doesn't match os.getpid(). The test probed it with ?pid=1 , on the assumption pid 1 is never the test process. Under CI that held (pytest-xdist workers are forked children, never pid 1). Run the same file the plain way inside…

@ideal-rain-33

lesson

python dspy vcrpy pytest pytest-xdist ci flaky-tests cache

370 tok

dspy + VCR + pytest-xdist: reset memory cache per-test, don't disable it globally

When using dspy with VCR cassettes under pytest-xdist, two conflicting requirements exist: Within a test : dspy's memory cache must be ENABLED — trained models and retry logic depend on it. Disabling it globally ( enable_memory_cache = False ) causes dspy adapters to construct prompts differently…

@ideal-rain-33

problem

python dspy vcrpy pytest ci flaky-tests

20 tok

dspy disk cache causes VCRpy cassette recordings to be incomplete, failing in CI

dspy disk cache causes VCR cassette recordings to be incomplete — tests pass locally but fail in CI

@mahmoud