Fedora impact check with 7.0.0rc1

[hroncok]

Hello, I am slowly triaging the impact of pytest 7.0.0rc1 on Fedora packages. Will cross post upstream issues here.

See #9415 (comment) for a full list of cross-references.

[bluetech]

@hroncok Great as always!

I will try to help with triage/fixes over the weekend.

Right now I only clicked bandit one: says ModuleNotFoundError: No module named 'toml' - it might be caused by pytest switching from toml to tomli and now the dep is missing. Probably needs an extra toml test dep (I do see it present in the test-requirements.txt file so maybe just missing in Fedora package).

[hroncok]

bandit in Fedora does not seem to install from https://github.com/PyCQA/bandit/blob/1.7.1/test-requirements.txt

[hroncok]

https://src.fedoraproject.org/rpms/bandit/pull-request/2

[The-Compiler]

Initial triage

Note: This post will be edited once the mentioned issues are "dealt with", i.e. the issues are triaged.

From our perspective, that means that there is either:

• We opened an issue in the pytest repository for it
• or the issue has been reported upstream (ideally via a PR!) AND we're confident there is nothing pytest needs to do
• or we deem the issue a "wontfix" (e.g. dead upstream) AND see above

I took a quick look through all the logs, just to get a quick overview. Here is what I found:

Packaging issues (Fedora specific)

Missing toml dependency (got pulled in via pytest before, which uses tomli now)

• bandit: fixed via https://src.fedoraproject.org/rpms/bandit/pull-request/2
• python-box: https://src.fedoraproject.org/rpms/python-box/pull-request/1
• python-vulture: https://src.fedoraproject.org/rpms/python-vulture/pull-request/1
• python-pytest-rerunfailures: E AttributeError: 'list' object has no attribute 'items' in for col, (finalizers, exc) in self.stack.items():
  Can't seem to consistently reproduce - I was bisecting it and suddenly it's gone even with pytest 7 now
  Already fixed in Fix SetupState breakage with upcoming pytest 6.3 pytest-rerunfailures#151, which was released in pytest-rerunfailures 10.0 back in May, but Fedora still seems to be at 9.1.1

Fatal warnings or wrong usage

• python-diff-cover: approx in boolean context, fixed via Fix tests for pytest 7.0 Bachmann1234/diff_cover#264
• python-ratelimiter: E pytest.PytestRemovedIn7Warning: pytest.collect.File was moved to pytest.File - probably not worth trying to fix, it's related to code for Python < 3.5 and no upstream commits in over 4 years despite open issues/PRs
• python-httplib2: Skip.__init__() got multiple values for argument 'reason' - maybe you meant pytest.mark.skipif? - wrong API usage: Fix incorrect pytest.mark.skip usage httplib2/httplib2#215

Various warns(None) usage, see #9002:

• python-aiohttp: warns(None)
• python-trio: warns(None)
• python-flask: warns(None)
• python-ase: warns(None)
• python-dateutil: warns(None)

Private API usage

• pg_activity: from _pytest.tmpdir import TempdirFactory via pytest_postgresql
• python-astroscrappy: E TypeError: Config._getconftest_pathlist() missing 1 required positional argument: 'rootpath' (collect_ignore = config._getconftest_pathlist("collect_ignore", path=dirpath))
  Happened via pytest-doctestplus, already fixed in its latest releases and newer Fedora tests seem to succeed too
• black: ImportError: Error importing plugin "tests.optional": No module named '_pytest.store' (reported upstream: Support pytest 7 psf/black#2705)
• python-pytest-cases: from _pytest.fixtures import scopes: Refactored scope handling in pytest 7 smarie/python-pytest-cases#241
• python-fiat: from _pytest.python import scopes via pytest-cases
• python-pytest-postgresql: from _pytest.tmpdir import TempdirFactory: Breaks with pytest 7 due to private imports dbfixtures/pytest-postgresql#533

Incorrect version parsing

• python-mirrors-countme: E ValueError: invalid literal for int() with base 10: '0rc1': Issue #57: Usage of pytest.version breaks testing with prereleases - mirrors-countme - Pagure.io
• python-pytest-isort: E ValueError: invalid literal for int() with base 10: '0rc1': get_pytest_version version parsing breaks with prereleases · Issue #38 · stephrdev/pytest-isort

Possible pytest regressions

• python-html5lib: Using Config.rootdir from pytest_configure (fix in progress in pytest: Ensure Config.inifile is available during pytest_cmdline_main #9400)
• python-promise: Accesses assert assert_exc.traceback[-1].path.strpath == __file__ (from ExceptionInfo), but path is a PosixPath now. Sounds like we falsely assumed that was internal only? PR open: Fix tests for pytest 7.0 compatibility syrusakbary/promise#96; pytest issue: [prerelease] ExceptionInfo.traceback[...].path changed from py.path to pathlib.Path #9423
• python-pytest-randomly: E AttributeError: 'Module' object has no attribute 'instance' - Test on pytest 7 pytest-randomly#423 and [prerelease] Removed instance attribute on Module / Class  #9486
• python-pytest-bdd: pytester test fails because more tests than expected are deselected
  Bisected to 6d6cfd8. Was this just not tested before? See Fix "deselect" keyword usage by pytester.assert_outcomes for pytest>6 pytest-bdd#470 and [prerelease] deselected addition to assert_outcomes() is backwards-incompatible #9471
• pytest-isort: Fails with if session is None: assert parent is None in nodes.py after Avoid brittle version parsing stephrdev/pytest-isort#39
  See [prerelease] pytest-isort and if session is None: assert parent is None #9487

Unclear

• ansible: No idea, most tests seem to fail due to a sys.exit apparently? Upstream: pytest 7 test failures ansible/ansible#76577.
  Skipped for now, could not get tests to run.
• ansible-core: Various failures with TypeError: mock_NamedTemporaryFile() missing 1 required positional argument: 'mocker'.
  Skipped for now, could not get tests to run.
• poetry: Reported upstream: pytest 7 test failures python-poetry/poetry#4901.
  Skipped for now, can't seem to reproduce locally.
• python-interrogate: ConftestImportFailure, can't see why.
  Bisected to 614f539 but I don't understand. PR sent - not pytest issue.
• python-josepy: E ModuleNotFoundError: No module named 'josepy'
  Can't seem to reproduce locally, had some issues with pytest-flake8 but works otherwise (via tox).
• python-parameterized: Unit test fails
  Breaks for me locally even on pytest 6 in the same way, last reference in tox.ini is to pytest 3. Also see Pytest 'yield tests are deprecated, and scheduled to be removed in pytest 4.0' wolever/parameterized#34.
• python-sqlalchemy1.3: "No tests ran": https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3332
• python-sibyl: Some AssertionError inside exec
  Bisected to 92ba96b but I don't understand why... Fixed in code: accept any os.PathLike[str] in Traceback.cut #9447

[hroncok]

I've added them to the Fedora list: https://src.fedoraproject.org/rpms/pytest/pull-request/25#comment-92228

• bandit
• python-box
• python-vulture

[nicoddemus]

Thanks! Added those to Florian's post. 👍

[hroncok]

I've also edited the headline to indicate the packaging issues are Fedora specific, not upstream packaging related.

[bluetech]

Sent a PR for interrogate: econchick/interrogate#99 I'd say it's not a pytest issue/regression, there are probably some existing issues about better recovery from monkeypatch patching stdlib modules but it's nothing new except that we use pathlib more.

[bluetech]

See https://github.com/pytest-dev/pytest/pull/9440/files#r775133826 for sybil

[musicinmybrain]

I took a look at python-pytest-bdd. A nuance is that it seems that more tests than expected are deselected, not in the actual test suite, but in the test suites that run run as part of certain individual tests. In fact, I just added a manual BR on python3dist(pytest-xdist) so that now no top-level tests are skipped (instead of one).

As downstream maintainer, I don’t really know how to fix these failures with a reasonable amount of investigative effort, so guess I’ll report them upstream and skip the affected tests for now.

[musicinmybrain]

I have reported the python-pytest-bdd failures upstream and skipped the affected tests, so it now builds in the COPR.

[kalekundert]

I fixed the bug in python-diff-cover: Bachmann1234/diff_cover#264

[The-Compiler]

FYI @hroncok I looked at some more:

• python-ratelimiter: E pytest.PytestRemovedIn7Warning: pytest.collect.File was moved to pytest.File - probably not worth trying to fix, it's related to code for Python < 3.5 and no upstream commits in over 4 years despite open issues/PRs
• pg_activity: from _pytest.tmpdir import TempdirFactory via pytest_postgresql
• python-pytest-cases: from _pytest.fixtures import scopes: Refactored scope handling in pytest 7 smarie/python-pytest-cases#241 (no patch, fix seems a bit more involved)
• python-fiat: from _pytest.python import scopes via pytest-cases
• python-pytest-postgresql: from _pytest.tmpdir import TempdirFactory: Breaks with pytest 7 due to private imports dbfixtures/pytest-postgresql#533 (trivial patch in comments, though a proper fix would do a bit more)

[The-Compiler]

Remaining issues

Since almost all failures are dealt with and the post above got a bit unwieldy, I'm starting a new one for the remaining ones:

• python-pytest-rerunfailures: E AttributeError: 'list' object has no attribute 'items' in for col, (finalizers, exc) in self.stack.items():
  Already fixed in Fix SetupState breakage with upcoming pytest 6.3 pytest-rerunfailures#151, which was released in pytest-rerunfailures 10.0 back in May, but Fedora still seems to be at 9.1.1 (cc @hroncok)
• pytest-isort fails with assert parent is not None: Unclear, asking for help in [prerelease] pytest-isort and if session is None: assert parent is None #9487
• pytest-randomly: Mainly worried about the undocumented instance attribute removal, [prerelease] Removed instance attribute on Module / Class  #9486
• josepy: Still unclear why it fails with a ModuleNotFoundError for the module under test on Fedora. I can't reproduce locally, so this might in fact be some kind of Fedora packaging issue (though I can't explain why it would work on pytest 6). Giving up.
• ansible: No idea, most tests seem to fail due to a sys.exit apparently? Upstream: pytest 7 test failures ansible/ansible#76577.
  Skipped for now, could not get tests to run.
• ansible-core: Various failures with TypeError: mock_NamedTemporaryFile() missing 1 required positional argument: 'mocker'.
  Skipped for now, could not get tests to run.
• poetry: Reported upstream: pytest 7 test failures python-poetry/poetry#4901.
  Can't reproduce even when trying without an internet connection. Note that I'm not entirely sure if that's really the issue - while there are warnings indicating network requests, the actual test failures might be from a different problem. Anyways, giving up on this one.
• ansible-runner: New failure in second Fedora run only, might just be a flaky test.

Taking another look at Ansible and Poetry now - for the others (and possibly those too), I don't think there is much more we can do from pytest's side.

[musicinmybrain]

* [ ]  `josepy`: Still unclear why it [fails](https://download.copr.fedorainfracloud.org/results/churchyard/pytest-7/fedora-rawhide-x86_64/03128647-python-josepy/builder-live.log.gz) with a `ModuleNotFoundError` for the module under test on Fedora. I can't reproduce locally, so this might in fact be some kind of Fedora packaging issue (though I can't explain why it would work on pytest 6).

I just submitted a PR for this: https://src.fedoraproject.org/rpms/python-josepy/pull-request/1

I’m not sure why setting PYTHONPATH became necessary where it was not before, but it’s a trivial fix in the Fedora packaging for josepy.

Here’s the successful COPR build for the PR: https://copr.fedorainfracloud.org/coprs/churchyard/pytest-7/build/3132715/

[The-Compiler]

I'm afraid I'll give up with Ansible as well. I get dozens of failing tests with pytest 6, and at least some of the thousands of failures with pytest 7 seem to be wrong API usage. To make things worse, something in their testsuite seems to change node IDs so it's a pain to even find the responsible tests...

Spent way too much time on this one now. Time for weekend instead!

[hroncok]

ansible-runner: New failure in second Fedora run only, might just be a flaky test.

I've submitted 10 builds with each pytest: https://copr.fedorainfracloud.org/coprs/churchyard/pytest-7/package/python-ansible-runner/ and https://copr.fedorainfracloud.org/coprs/churchyard/pytest-6/package/python-ansible-runner/

[hroncok]

python-pytest-rerunfailures: E AttributeError: 'list' object has no attribute 'items' in for col, (finalizers, exc) in self.stack.items():
Already fixed in pytest-dev/pytest-rerunfailures#151, which was released in pytest-rerunfailures 10.0 back in May, but Fedora still seems to be at 9.1.1 (cc @hroncok)

I've noted this in the python-pytest-rerunfailures update bugzilla and linked it to the pytest update bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1965106#c3

[The-Compiler]

Looks like the ansible-runner test is indeed flaky: 2/10 fail with pytest 6 (same issue), 5/10 with pytest 7. Looks like there's Clear test flake (test_password_prompt) by jladdjr · Pull Request #472 · ansible/ansible-runner (should be already in with the tested version), but perhaps that's not a full fix.