Test re-ordering breaks Pytest's --failed-first and --stepwise options

[callahad]

The test re-ordering introduced in response to #214 seems to execute after pytest's own --failed-first, --new-first, --stepwise, etc. ordering options, breaking them.

We ran across this in mdn/kuma#6531, where even with --failed-first pytest was running dozens of known good tests before executing the failed tests.

Removing the pytest_collection_modifyitems function or decorating it with @pytest.hookimpl(tryfirst=True) seems to resolve this, though I'm not familiar enough with pytest to know if that's an appropriate solution, or if there's something we should be doing on our end instead.

[callahad]

The function in question can be found at https://github.com/pytest-dev/pytest-django/blob/v3.8.0/pytest_django/plugin.py#L418-L439, and was added in commit 5a79fba

[blueyed]

Thanks for the report.
Can you try it with pytest master? IIRC I've fixed something there in this regard.

[callahad]

Thank you for the quick response!

I see the same issue with pytest-5.3.5.dev399+gf77d606d4, which is the current head of master

pytest --ff -x kuma/users/tests/test_views.py                                                                                                    
=== test session starts ===
platform linux -- Python 3.8.0, pytest-5.3.5.dev399+gf77d606d4, py-1.8.1, pluggy-0.13.1
django: settings: kuma.settings.testing (from ini)
rootdir: /app, inifile: pytest.ini
plugins: base-url-1.4.1, celery-4.4.0, django-3.8.0, metadata-1.8.0, requests-mock-1.7.0, cov-2.8.1, rerunfailures-8.0
collected 72 items                                                                                                                                                                                         
run-last-failure: rerun previous 1 failure first

kuma/users/tests/test_views.py ............................F

[blueyed]

Ok. Using @pytest.hookimpl(tryfirst=True) then seems to be sensible.
Would you like to create a PR for this?

(I was thinking of pytest-dev/pytest@f75f7c1, which fixed it for pytest's own tests)

[callahad]

Sure! Pull request at #820