`tmp_path`-fixtured tests generally do not expect neither parallelism nor repetition

[ogrisel]

Consider a test file named test_with_tmp_path_fixture.py that uses the tmp_path fixture:

def test_with_tmp_path(tmp_path):
    assert tmp_path.exists()
    assert tmp_path.is_dir()
    assert len(list(tmp_path.iterdir())) == 0
    d = tmp_path / "sub"
    assert not d.exists()
    d.mkdir()
    assert d.exists()

It's quite typical from such tests to assume that the folder generated by the fixture is empty when executed.

Running such a test with pytest-run-parallel will therefore fail either because of parallelism or repetition:

• parallelism (one execution per thread):

pytest -v --parallel-threads=4 --iterations=1 test_with_tmp_path_fixture.py

================================================================= test session starts ==================================================================
platform darwin -- Python 3.13.3, pytest-8.4.0, pluggy-1.6.0 -- /Users/ogrisel/miniforge3/envs/dev/bin/python3.13
cachedir: .pytest_cache
rootdir: /Users/ogrisel/tmp
plugins: run-parallel-0.6.1.dev0, xdist-3.7.0, anyio-4.9.0
collected 1 item                                                                                                                                       
Collected 1 items to run in parallel

test_with_tmp_path_fixture.py::test_with_tmp_path PARALLEL FAILED                                                                                [100%]

======================================================================== ERRORS ========================================================================
_________________________________________________________ ERROR at call of test_with_tmp_path __________________________________________________________

tmp_path = PosixPath('/private/var/folders/93/0_k_9bh97fj_15hzk2bhy5r00000gn/T/pytest-of-ogrisel/pytest-15/test_with_tmp_path0')

    def test_with_tmp_path(tmp_path):
        assert tmp_path.exists()
        assert tmp_path.is_dir()
        assert len(list(tmp_path.iterdir())) == 0
        d = tmp_path / "sub"
        assert not d.exists()
>       d.mkdir()

test_with_tmp_path_fixture.py:7: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('/private/var/folders/93/0_k_9bh97fj_15hzk2bhy5r00000gn/T/pytest-of-ogrisel/pytest-15/test_with_tmp_path0/sub'), mode = 511
parents = False, exist_ok = False

    def mkdir(self, mode=0o777, parents=False, exist_ok=False):
        """
        Create a new directory at this given path.
        """
        try:
>           os.mkdir(self, mode)
E           FileExistsError: [Errno 17] File exists: '/private/var/folders/93/0_k_9bh97fj_15hzk2bhy5r00000gn/T/pytest-of-ogrisel/pytest-15/test_with_tmp_path0/sub'

../miniforge3/envs/dev/lib/python3.13/pathlib/_local.py:722: FileExistsError
************************************************************** pytest-run-parallel report **************************************************************
All tests were run in parallel! 🎉
=============================================================== short test summary info ================================================================
PARALLEL FAILED test_with_tmp_path_fixture.py::test_with_tmp_path - FileExistsError: [Errno 17] File exists: '/private/var/folders/93/0_k_9bh97fj_15hzk2bhy5r00000gn/T/pytest-of-ogrisel/pytest-15/test_with_tmp_path0/...
=================================================================== 1 error in 0.10s ===================================================================

• sequential repetition:

pytest -v --parallel-threads=1 --iterations=2 test_with_tmp_path_fixture.py

================================================================= test session starts ==================================================================
platform darwin -- Python 3.13.3, pytest-8.4.0, pluggy-1.6.0 -- /Users/ogrisel/miniforge3/envs/dev/bin/python3.13
cachedir: .pytest_cache
rootdir: /Users/ogrisel/tmp
plugins: run-parallel-0.6.1.dev0, xdist-3.7.0, anyio-4.9.0
collected 1 item                                                                                                                                       
Collected 0 items to run in parallel

test_with_tmp_path_fixture.py::test_with_tmp_path FAILED                                                                                         [100%]

======================================================================= FAILURES =======================================================================
__________________________________________________________________ test_with_tmp_path __________________________________________________________________

tmp_path = PosixPath('/private/var/folders/93/0_k_9bh97fj_15hzk2bhy5r00000gn/T/pytest-of-ogrisel/pytest-16/test_with_tmp_path0')

    def test_with_tmp_path(tmp_path):
        assert tmp_path.exists()
        assert tmp_path.is_dir()
>       assert len(list(tmp_path.iterdir())) == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = len([PosixPath('/private/var/folders/93/0_k_9bh97fj_15hzk2bhy5r00000gn/T/pytest-of-ogrisel/pytest-16/test_with_tmp_path0/sub')])
E        +    where [PosixPath('/private/var/folders/93/0_k_9bh97fj_15hzk2bhy5r00000gn/T/pytest-of-ogrisel/pytest-16/test_with_tmp_path0/sub')] = list(<map object at 0x10c9704f0>)
E        +      where <map object at 0x10c9704f0> = iterdir()
E        +        where iterdir = PosixPath('/private/var/folders/93/0_k_9bh97fj_15hzk2bhy5r00000gn/T/pytest-of-ogrisel/pytest-16/test_with_tmp_path0').iterdir

test_with_tmp_path_fixture.py:4: AssertionError
=============================================================== short test summary info ================================================================
FAILED test_with_tmp_path_fixture.py::test_with_tmp_path - AssertionError: assert 1 == 0
================================================================== 1 failed in 0.35s ===================================================================

but this tests does pass as expected when disabling both parallelism and repetition together:

pytest -v --parallel-threads=1 --iterations=1 test_with_tmp_path_fixture.py

================================================================= test session starts ==================================================================
platform darwin -- Python 3.13.3, pytest-8.4.0, pluggy-1.6.0 -- /Users/ogrisel/miniforge3/envs/dev/bin/python3.13
cachedir: .pytest_cache
rootdir: /Users/ogrisel/tmp
plugins: run-parallel-0.6.1.dev0, xdist-3.7.0, anyio-4.9.0
collected 1 item                                                                                                                                       
Collected 0 items to run in parallel

test_with_tmp_path_fixture.py::test_with_tmp_path PASSED                                                                                         [100%]

================================================================== 1 passed in 0.09s ===================================================================

Note that it is now possible to configure pyproject.toml as follows:

[tool.pytest.ini_options]
thread_unsafe_fixtures = [
  "tmp_path",  # does not isolate temporary directories across threads.
]

This configuration provides a low-maintenance solution to the first problem (maybe this should be the default?). However, as far as I know, there is no equivalent configuration to automatically mark those tests as repetition-unsafe.

Final note: there are other fixtures with the same problem, for instance tmpdir.

[ngoldbaum]

Maybe should add these fixtures to the default thread-unsafe fixtures list.

The problem is you can update tests to use these fixtures safely and it would be a shame IMO if tests we've already updated suddenly started getting marked as thread-unsafe due to a change in pytest-run-parallel.

Maybe we could add new thread-safe versions of these fixtures that people can use instead? We could also suggest that there is a thread-safe alternative when we know there is one in the message explaining why a test was marked as unsafe.

I'm also a little curious what would happen if we rewrote the plugin following my suggestion here but unfortunately don't have time to explore that right now and the other maintainer is about to go on vacation for a few weeks...

[ngoldbaum]

However, as far as I know, there is no equivalent configuration to automatically mark those tests as repetition-unsafe.

This issue hasn't come up yet, I think it's worth adding an analogous config knob for repetitions to fix this.

Or maybe just don't allow repetitions of tests marked as thread-unsafe, for simplicity?

[lysnikolaou]

Or maybe just don't allow repetitions of tests marked as thread-unsafe, for simplicity?

Maybe even go as far as disallowing repetitions altogether for tests that'll run in a single thread, meaning even for cases where the tests weren't automatically marked but someone used --parallel-threads=1. Using --parallel-threads=1 --iterations=>1 sounds like something people should be using pytest-repeat for.

[ogrisel]

I agree with the above suggestions.

[ngoldbaum]

I'm working with an intern this summer on running pytest-run-parallel in the NumPy CI. I've asked her (@bwhitt7) to take a look at this issue, since NumPy also uses tmp_path.

[ngoldbaum]

@ogrisel I think #120 fixed this, at least for parallelism. Does that seem reasonable to you?

[ogrisel]

Indeed the parallelism problem is fixed but not the repetition problem. Maybe a config option to disable repetition for some fixtures would be good?

[ngoldbaum]

I think a repetition_unsafe marker is reasonable to add. You can also write the test so the paths that get created have to be unique (e.g. with a uuid or some other random extra data) but I could see a one-liner being nice.

Another option is a command-line option to make these fixtures repetition-safe.

We ended up not always making them repetition-safe because creating and removing so many extra directories added a ~10-15% runtime cost for running a the scipy io tests with repetitions (I'd need to dig to find the exact numbers). We didn't want to make everyone pay that cost but we can add a flag to let people opt into it. We do already have a lot of flags, though.