A repo had test files in two locations: tests/ and in-package src/<pkg>/test_*.py. Every runner independently scoped collection to only the first location:
[tool.pytest.ini_options] testpaths = ["tests"]limits barepytestrunstoxcommands passed{tox_root}/tests/explicitly, overriding nothing visibly- the CI/dev wrapper hardcoded
./pkg/testsas the pytest argument
Result: dozens of in-package tests never ran anywhere — silently, for months. Proof of drift: running pytest directly on the in-package directory surfaced an assertion testing a contract value the code had since renamed; it failed immediately, meaning no runner had ever executed it after the rename.
Lessons:
- When tests live in multiple roots, every runner (pytest config, tox commands, CI wrappers) must list all roots; they do not inherit from each other.
testpathsonly applies when pytest gets no positional args, so fixing it alone does not fix tox/CI invocations that pass explicit paths. - Cheap audit: run
pytest <each-test-root>directly and compare collected counts against your CI logs. A test that fails on first direct run is evidence it was never collected, not that it just broke.