pytest keeps running deleted tests

[pytestbot]

Originally reported by: Mantas Zilinskis (BitBucket: montiniz, GitHub: montiniz)

---

i just started using pytest for my django projects.
couple days ago playing around with a test project i noticed that pytest sometimes would run some test functions which were already deleted. did not pay attention to that then.
today i started new django project with pytest-bdd, pytest-django. after creating some test function, ran tests. then decided to remove one test function. ran the test again and noticed that pytest collected the deleted test.
removed all pyc files from working direcotry, cleared all pycahe directories, ran tests and it still collected the deleted test. whatever i tried wouldn't help. i'm running on virtualenv, python 2.7, django 1.6., MacOSX 10.9.1, SublimeText 3beta.
what would be the way to debug this?

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/471

[pytestbot]

Original comment by Jurko Gospodnetić (BitBucket: jurko, GitHub: jurko):

---

Have you tried running pytest with the --collect-only command-line option? That should tell you the exact list of tests collected, including all related information - module, test class, test class instance, test method/function.

I do not think pytest makes a copy of your tests anywhere 😄 so more than likely you simply forgot to remove whatever location pytest is collecting those tests from.

For example, I occasionally remove a test module, rerun a py2to3 transformation and then run py3 tests, only to find the removed tests still being run. This happens because py2to3 converts existing py2 modules to py3 in some build folder but does not remove existing py3 modules in that build folder for which there is no longer a corresponding py2 module. A simple fix for my situation is to manually remove the build folder when something like that occurs.

Hope this helps.

Best regards,
Jurko Gospodnetić

[pytestbot]

Original comment by Jurko Gospodnetić (BitBucket: jurko, GitHub: jurko):

---

@montiniz - you still following this issue? Could you give some more information on it? Have you tried the --collect-only command-line option?

Best regards,
Jurko Gospodnetić

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

pytest won't load tests from pyc files only -- it only collects ones that have a .py files. And indeed, pytest itself certainly does not copy files. Please check your environment and what directories are actually collected like Jurko suggested. If anything, this is a case of coming to IRC (#pylib or #python) and asking for help there.

[George-Ogden]

This issue doesn't seem to have been fixed, and here is a minimal example to reproduce...
test_main.py:

import pytest
# import importlib

def test_nothing():
    ...

if __name__ == "__main__":
    pytest.main(["--co"])
    input("Comment out `test_nothing()`")
    # importlib.reload(pytest)
    pytest.main(["--co"])

Midway through, comment out the test_nothing function - it still lists the deleted test. (Some further experimentation shows that it will actually run the deleted test too.)

Even when I reload pytest (introducing commented-out code), I get the same result. But if I start with the function commented-out and then uncomment it, I go from zero tests to one.

[The-Compiler]

This is expected. Your file is already imported and then pytest runs on the imported module. If anything you'd need to reload your own code (not pytest), but reloading Python code comes with a lot of gotchas, so my best advice is along the lines of "just don't do this".

[George-Ogden]

@The-Compiler, thanks for your response. I agree that it's an unusual use case, but can you explain/hint at why this happens? (No worries if it's too complex to put in a single comment.)

[The-Compiler]

Consider this for starters:

import sys
import inspect


def test_nothing():
    pass


if __name__ == "__main__":
    mod = sys.modules["__main__"]
    print(inspect.getmembers(mod, inspect.isfunction))
    input("Comment out `test_nothing()`")
    print(inspect.getmembers(mod, inspect.isfunction))

It's not quite the same situation as with pytest, but close enough to demonstrate why this wouldn't work. With this snippet, importlib.reload won't work, but you can simulate something similar by moving the def test_nothing(): pass to a x2.py and doing:

import sys
import inspect
import importlib

import x2


if __name__ == "__main__":
    print(inspect.getmembers(x2, inspect.isfunction))
    input("Comment out `test_nothing()`")
    importlib.reload(x2)
    print(inspect.getmembers(x2, inspect.isfunction))

As the importlib.reload docs point out:

When a module is reloaded, its dictionary (containing the module’s global variables) is retained. [...] If the new version of a module does not define a name that was defined by the old version, the old definition remains.