Skip to content

Commit 0e62302

Browse files
bluetechbluetech
committed
Rework Session and Package collection
Fix #7777.
1 parent 0c32c88 commit 0e62302

33 files changed

+865
-333
lines changed

‎changelog/7777.breaking.rst

+87
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
Added a new :class:`pytest.Directory` base collection node, which all collector nodes for filesystem directories are expected to subclass.
2+
This is analogous to the existing :class:`pytest.File` for file nodes.
3+
4+
Changed :class:`pytest.Package` to be a subclass of :class:`pytest.Directory`.
5+
A ``Package`` represents a filesystem directory which is a Python package,
6+
i.e. contains an ``__init__.py`` file.
7+
8+
:class:`pytest.Package` now only collects files in its own directory; previously it collected recursively.
9+
Sub-directories are collected as sub-collector nodes, thus creating a collection tree which mirrors the filesystem hierarchy.
10+
11+
Added a new :class:`pytest.Dir` concrete collection node, a subclass of :class:`pytest.Directory`.
12+
This node represents a filesystem directory, which is not a :class:`pytest.Package`,
13+
i.e. does not contain an ``__init__.py`` file.
14+
Similarly to ``Package``, it only collects the files in its own directory,
15+
while collecting sub-directories as sub-collector nodes.
16+
17+
Added a new hook :hook:`pytest_collect_directory`,
18+
which is called by filesystem-traversing collector nodes,
19+
such as :class:`pytest.Session`, :class:`pytest.Dir` and :class:`pytest.Package`,
20+
to create a collector node for a sub-directory.
21+
It is expected to return a subclass of :class:`pytest.Directory`.
22+
This hook allows plugins to :ref:`customize the collection of directories <custom directory collectors>`.
23+
24+
:class:`pytest.Session` now only collects the initial arguments, without recursing into directories.
25+
This work is now done by the :func:`recursive expansion process <pytest.Collector.collect>` of directory collector nodes.
26+
27+
:attr:`session.name <pytest.Session.name>` is now ``""``; previously it was the rootdir directory name.
28+
This matches :attr:`session.nodeid <_pytest.nodes.Node.nodeid>` which has always been `""`.
29+
30+
Files and directories are now collected in alphabetical order jointly, unless changed by a plugin.
31+
Previously, files were collected before directories.
32+
33+
The collection tree now contains directories/packages up to the :ref:`rootdir <rootdir>`,
34+
for initial arguments that are found within the rootdir.
35+
For files outside the rootdir, only the immediate directory/package is collected (this is discouraged).
36+
37+
As an example, given the following filesystem tree::
38+
39+
myroot/
40+
pytest.ini
41+
top/
42+
├── aaa
43+
│ └── test_aaa.py
44+
├── test_a.py
45+
├── test_b
46+
│ ├── __init__.py
47+
│ └── test_b.py
48+
├── test_c.py
49+
└── zzz
50+
├── __init__.py
51+
└── test_zzz.py
52+
53+
the collection tree, as shown by `pytest --collect-only top/` but with the otherwise-hidden :class:`~pytest.Session` node added for clarity,
54+
is now the following::
55+
56+
<Session>
57+
<Dir myroot>
58+
<Dir top>
59+
<Dir aaa>
60+
<Module test_aaa.py>
61+
<Function test_it>
62+
<Module test_a.py>
63+
<Function test_it>
64+
<Package test_b>
65+
<Module test_b.py>
66+
<Function test_it>
67+
<Module test_c.py>
68+
<Function test_it>
69+
<Package zzz>
70+
<Module test_zzz.py>
71+
<Function test_it>
72+
73+
Previously, it was::
74+
75+
<Session>
76+
<Module top/test_a.py>
77+
<Function test_it>
78+
<Module top/test_c.py>
79+
<Function test_it>
80+
<Module top/aaa/test_aaa.py>
81+
<Function test_it>
82+
<Package test_b>
83+
<Module test_b.py>
84+
<Function test_it>
85+
<Package zzz>
86+
<Module test_zzz.py>
87+
<Function test_it>

‎doc/en/deprecations.rst

+83
Original file line numberDiff line numberDiff line change
@@ -495,6 +495,89 @@ an appropriate period of deprecation has passed.
495495
Some breaking changes which could not be deprecated are also listed.
496496

497497

498+
Collection changes in pytest 8
499+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
500+
501+
Added a new :class:`pytest.Directory` base collection node, which all collector nodes for filesystem directories are expected to subclass.
502+
This is analogous to the existing :class:`pytest.File` for file nodes.
503+
504+
Changed :class:`pytest.Package` to be a subclass of :class:`pytest.Directory`.
505+
A ``Package`` represents a filesystem directory which is a Python package,
506+
i.e. contains an ``__init__.py`` file.
507+
508+
:class:`pytest.Package` now only collects files in its own directory; previously it collected recursively.
509+
Sub-directories are collected as sub-collector nodes, thus creating a collection tree which mirrors the filesystem hierarchy.
510+
511+
:attr:`session.name <pytest.Session.name>` is now ``""``; previously it was the rootdir directory name.
512+
This matches :attr:`session.nodeid <_pytest.nodes.Node.nodeid>` which has always been `""`.
513+
514+
Added a new :class:`pytest.Dir` concrete collection node, a subclass of :class:`pytest.Directory`.
515+
This node represents a filesystem directory, which is not a :class:`pytest.Package`,
516+
i.e. does not contain an ``__init__.py`` file.
517+
Similarly to ``Package``, it only collects the files in its own directory,
518+
while collecting sub-directories as sub-collector nodes.
519+
520+
Files and directories are now collected in alphabetical order jointly, unless changed by a plugin.
521+
Previously, files were collected before directories.
522+
523+
The collection tree now contains directories/packages up to the :ref:`rootdir <rootdir>`,
524+
for initial arguments that are found within the rootdir.
525+
For files outside the rootdir, only the immediate directory/package is collected (this is discouraged).
526+
527+
As an example, given the following filesystem tree::
528+
529+
myroot/
530+
pytest.ini
531+
top/
532+
├── aaa
533+
│ └── test_aaa.py
534+
├── test_a.py
535+
├── test_b
536+
│ ├── __init__.py
537+
│ └── test_b.py
538+
├── test_c.py
539+
└── zzz
540+
├── __init__.py
541+
└── test_zzz.py
542+
543+
the collection tree, as shown by `pytest --collect-only top/` but with the otherwise-hidden :class:`~pytest.Session` node added for clarity,
544+
is now the following::
545+
546+
<Session>
547+
<Dir myroot>
548+
<Dir top>
549+
<Dir aaa>
550+
<Module test_aaa.py>
551+
<Function test_it>
552+
<Module test_a.py>
553+
<Function test_it>
554+
<Package test_b>
555+
<Module test_b.py>
556+
<Function test_it>
557+
<Module test_c.py>
558+
<Function test_it>
559+
<Package zzz>
560+
<Module test_zzz.py>
561+
<Function test_it>
562+
563+
Previously, it was::
564+
565+
<Session>
566+
<Module top/test_a.py>
567+
<Function test_it>
568+
<Module top/test_c.py>
569+
<Function test_it>
570+
<Module top/aaa/test_aaa.py>
571+
<Function test_it>
572+
<Package test_b>
573+
<Module test_b.py>
574+
<Function test_it>
575+
<Package zzz>
576+
<Module test_zzz.py>
577+
<Function test_it>
578+
579+
580+
498581
:class:`pytest.Package` is no longer a :class:`pytest.Module` or :class:`pytest.File`
499582
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
500583

‎doc/en/example/conftest.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
collect_ignore = ["nonpython"]
1+
collect_ignore = ["nonpython", "customdirectory"]

‎doc/en/example/customdirectory.rst

+78
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
.. _`custom directory collectors`:
2+
3+
Using a custom directory collector
4+
====================================================
5+
6+
By default, pytest collects directories using :class:`pytest.Package`, for directories with ``__init__.py`` files,
7+
and :class:`pytest.Dir` for other directories.
8+
If you want to customize how a directory is collected, you can write your own :class:`pytest.Directory` collector,
9+
and use :hook:`pytest_collect_directory` to hook it up.
10+
11+
.. _`directory manifest plugin`:
12+
13+
A basic example for a directory manifest file
14+
--------------------------------------------------------------
15+
16+
Suppose you want to customize how collection is done on a per-directory basis.
17+
Here is an example ``conftest.py`` plugin.
18+
This plugin allows directories to contain a ``manifest.json`` file,
19+
which defines how the collection should be done for the directory.
20+
In this example, only a simple list of files is supported,
21+
however you can imagine adding other keys, such as exclusions and globs.
22+
23+
.. include:: customdirectory/conftest.py
24+
:literal:
25+
26+
You can create a ``manifest.json`` file and some test files:
27+
28+
.. include:: customdirectory/tests/manifest.json
29+
:literal:
30+
31+
.. include:: customdirectory/tests/test_first.py
32+
:literal:
33+
34+
.. include:: customdirectory/tests/test_second.py
35+
:literal:
36+
37+
.. include:: customdirectory/tests/test_third.py
38+
:literal:
39+
40+
An you can now execute the test specification:
41+
42+
.. code-block:: pytest
43+
44+
customdirectory $ pytest
45+
=========================== test session starts ============================
46+
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y
47+
rootdir: /home/sweet/project/customdirectory
48+
configfile: pytest.ini
49+
collected 2 items
50+
51+
tests/test_first.py . [ 50%]
52+
tests/test_second.py . [100%]
53+
54+
============================ 2 passed in 0.12s =============================
55+
56+
.. regendoc:wipe
57+
58+
Notice how ``test_three.py`` was not executed, because it is not listed in the manifest.
59+
60+
You can verify that your custom collector appears in the collection tree:
61+
62+
.. code-block:: pytest
63+
64+
customdirectory $ pytest --collect-only
65+
=========================== test session starts ============================
66+
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y
67+
rootdir: /home/sweet/project/customdirectory
68+
configfile: pytest.ini
69+
collected 2 items
70+
71+
<Dir customdirectory>
72+
<ManifestDirectory tests>
73+
<Module test_first.py>
74+
<Function test_1>
75+
<Module test_second.py>
76+
<Function test_2>
77+
78+
======================== 2 tests collected in 0.12s ========================

‎doc/en/example/customdirectory/conftest.py

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# content of conftest.py
2+
import json
3+
4+
import pytest
5+
6+
7+
class ManifestDirectory(pytest.Directory):
8+
def collect(self):
9+
manifest_path = self.path / "manifest.json"
10+
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
11+
ihook = self.ihook
12+
for file in manifest["files"]:
13+
yield from ihook.pytest_collect_file(
14+
file_path=self.path / file, parent=self
15+
)
16+
17+
18+
@pytest.hookimpl
19+
def pytest_collect_directory(path, parent):
20+
if path.joinpath("manifest.json").is_file():
21+
return ManifestDirectory.from_parent(parent=parent, path=path)
22+
return None

‎doc/en/example/customdirectory/pytest.ini

Whitespace-only changes.

‎doc/en/example/customdirectory/tests/manifest.json

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"files": [
3+
"test_first.py",
4+
"test_second.py"
5+
]
6+
}

‎doc/en/example/customdirectory/tests/test_first.py

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# content of test_first.py
2+
def test_1():
3+
pass

‎doc/en/example/customdirectory/tests/test_second.py

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# content of test_second.py
2+
def test_2():
3+
pass

‎doc/en/example/customdirectory/tests/test_third.py

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# content of test_third.py
2+
def test_3():
3+
pass

‎doc/en/example/index.rst

+1
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,4 @@ The following examples aim at various use cases you might encounter.
3232
special
3333
pythoncollection
3434
nonpython
35+
customdirectory

‎doc/en/reference/reference.rst

+14
Original file line numberDiff line numberDiff line change
@@ -662,6 +662,8 @@ Collection hooks
662662
.. autofunction:: pytest_collection
663663
.. hook:: pytest_ignore_collect
664664
.. autofunction:: pytest_ignore_collect
665+
.. hook:: pytest_collect_directory
666+
.. autofunction:: pytest_collect_directory
665667
.. hook:: pytest_collect_file
666668
.. autofunction:: pytest_collect_file
667669
.. hook:: pytest_pycollect_makemodule
@@ -900,6 +902,18 @@ Config
900902
.. autoclass:: pytest.Config()
901903
:members:
902904

905+
Dir
906+
~~~
907+
908+
.. autoclass:: pytest.Dir()
909+
:members:
910+
911+
Directory
912+
~~~~~~~~~
913+
914+
.. autoclass:: pytest.Directory()
915+
:members:
916+
903917
ExceptionInfo
904918
~~~~~~~~~~~~~
905919

‎src/_pytest/cacheprovider.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,8 @@
2727
from _pytest.fixtures import fixture
2828
from _pytest.fixtures import FixtureRequest
2929
from _pytest.main import Session
30+
from _pytest.nodes import Directory
3031
from _pytest.nodes import File
31-
from _pytest.python import Package
3232
from _pytest.reports import TestReport
3333

3434
README_CONTENT = """\
@@ -222,7 +222,7 @@ def pytest_make_collect_report(
222222
self, collector: nodes.Collector
223223
) -> Generator[None, CollectReport, CollectReport]:
224224
res = yield
225-
if isinstance(collector, (Session, Package)):
225+
if isinstance(collector, (Session, Directory)):
226226
# Sort any lf-paths to the beginning.
227227
lf_paths = self.lfplugin._last_failed_paths
228228

‎src/_pytest/config/__init__.py

-2
Original file line numberDiff line numberDiff line change
@@ -415,8 +415,6 @@ def __init__(self) -> None:
415415
# session (#9478), often with the same path, so cache it.
416416
self._get_directory = lru_cache(256)(_get_directory)
417417

418-
self._duplicatepaths: Set[Path] = set()
419-
420418
# plugins that were explicitly skipped with pytest.skip
421419
# list of (module name, skip reason)
422420
# previously we would issue a warning when a plugin was skipped, but

‎src/_pytest/hookspec.py

+15
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,21 @@ def pytest_ignore_collect(
284284
"""
285285

286286

287+
@hookspec(firstresult=True)
288+
def pytest_collect_directory(path: Path, parent: "Collector") -> "Optional[Collector]":
289+
"""Create a :class:`~pytest.Collector` for the given directory, or None if
290+
not relevant.
291+
292+
.. versionadded:: 8.0
293+
294+
The new node needs to have the specified ``parent`` as a parent.
295+
296+
Stops at first non-None result, see :ref:`firstresult`.
297+
298+
:param path: The path to analyze.
299+
"""
300+
301+
287302
def pytest_collect_file(
288303
file_path: Path, path: "LEGACY_PATH", parent: "Collector"
289304
) -> "Optional[Collector]":

0 commit comments

Comments
 (0)