Diff pytest --durations across historical CI runs before blaming the commit that first went red

A test started failing on master and the obvious suspect was the commit on the failing run -- it touched exactly the module under test. Wrong suspect. The test was a wall-clock-deadline flake that had been drifting for days; the blamed commit merely happened to be the next push after it crossed the line.

The cheap discriminator: most CI suites already print --durations=N, so every historical run log carries a timing sample for the test. Pull the same line out of the last few runs and look at the trend before reading any diff.

# newest first; note the conclusion column
gh run list --limit 10

# same test, one line per historical run
for r in <run_id> <run_id> <run_id>; do
  echo "== $r"
  gh run view $r --log 2>/dev/null | grep -E 'test_name_here' | tail -3
done

Output read: 59.35s ... PASSED -> 120.17s ... PASSED (deadline 120s, passed by 0.17s) -> FAILED. A monotone climb across unrelated commits with a pass squeaking in just under a hardcoded bound is proof of drift, not of a code regression. Bisecting would have blamed an innocent commit and produced a bogus revert.

Two more practical notes from the same hunt:

gh run view --job <id> --log-failed gives the pytest short summary (FAILED test_x - AssertionError:..., truncated). For the actual traceback and the durations block, use --log and grep for FAILURES, AssertionError, and the test id.
Once the trend says drift, the fix is not a bigger deadline. Find what the runtime is proportional to (here: accumulated process heap x sibling CPU contention on a 4-vCPU runner with -n 4), then remove that coupling -- e.g. route the test to a lower-parallelism CI shard via a marker -- and size the bound for continued growth.