ImportMismatchError with files under .tox

[jaraco]

I'm attempting to convert my projects away from pytest-runner and instead use tox.

In jaraco.mongodb (tox), you can see the tests fail to collect due to repeated ImportMismatchErrors for every module in the package. This happens on every Python version. You can see the failed tests in the Travis Job, but I get the same errors locally when running simply tox.

As you can see in the code, .tox is indicated in norecursedirs.

What can I do next to investigate?

[RonnyPfannschmidt]

this is not a tox issue, but a pytest interaction issue

doctest-modules looks in the checkout
but you installed the modules

thus it breaks apart, because the imported installed module is different from the doctested ones

[jaraco]

So - should pytest doctest-modules honor norecursedirs? Or is there another possible workaround?

[RonnyPfannschmidt]

it honors norecursedirs

the problem is - your test imports the installed module thats inside the virtualenv

afterwards doctest-modules comes along and takes a look at the source modules trying to import them failing with a missmatch

[jaraco]

What does tox recommend in a scenario like this? Is it possible to do what pytest-runner does and install the source modules for running tests?

[RonnyPfannschmidt]

RhE usedevelop flag

[jaraco]

Yeah, I was afraid you were going to say that. Unfortunately the develop flag has the issue that it doesn't work with namespace packages. Fortunately, there's a pending patch to fix that. Maybe that will work.

[hpk42]

hum, i wonder if there should be a way to disable the import-mismatch error.

Also, what about comparing the contents of the source files and, if they are equal, allowing it, maybe with a warning?

[RonnyPfannschmidt]

from my pov its a fundamental bug casued by collection differences,

tests are collected in worktree
code is collected at install location,

this is a py.test issue where we should consider declaring collections (so it will look for doctest tests in the installed python modules from pythonpath, not random files in the worktree

i think it is permissible to just warn on import mismatch
but the fundamental problem is, that collected files are at the src tree and the imported modules are at the install tree, and pytest has not even the hint of a model to support that

[hpk42]

i understand the intention of the warning but i wonder if it is always valid. In a typical tox invocation, an sdist gets created and installed. The test files are then typically the same in the install-tree and the source-tree.

Anyway, @jaraco, can you maybe just execute pytest --pyargs jaraco.mongodb so that pytest will collect from the installed package?

[RonnyPfannschmidt]

@hpk42 doctest-modules does doctests on non-test files

[hpk42]

doctest-modules does doctests on non-test files but when we are specifying --pyargs jaraco.mongodb we are telling pytest to collect from the import path of that package. So test files or not, it should work better.

[jaraco]

I've been using the usedevelop feature, which aligns best with the pytest-runner model (running the code as found in the repo, rather than testing an installed copy), and that's worked well except where bugs exist for the develop command.

can you maybe just execute pytest --pyargs jaraco.mongodb?

Perhaps I could, and I did just try it to see if it would work, but specifying --pyargs appears to be incompatible with --ignore and one module needs to be ignored. I did try --ignore jaraco.mongodb.pmxbot, but pytest still attempted to collect it.

More importantly, this workaround won't work in general because it excludes tests that might have been discovered outside of the package (such as in ./tests), but also requires that each project that uses this technique needs to diverge in test invocation (each package has to specify its own package name). I'd much rather be able to rely on the discovery mechanism to discover tests and not have to manually direct the test collection.

So for my purposes, I think the usedevelop technique is the best one as it achieves what I'm after and avoids duplicating files.

However, since the default (and presumably recommended) behavior is to install the package and test it from the installed location, I do think it makes sense that test collection should actively ignore the source files that were installed. I'm not sure it's worth the investment and complication to make that happen. May I suggest that we close this ticket as 'wontfix' for now, and perhaps reopen it as further use-cases demand?