Adding --cov-fail-under=100 on adopts config will provoque discovery failure when coverage under 100%

[lferro9000]

Environment data

• VS Code version: 1.45.1
• Extension version (available under the Extensions sidebar): 2020.5.80290
• OS and version: macOS 10.15.4
• Python version (& distribution if applicable, e.g. Anaconda): homebrew 3.7.7
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
• Relevant/affected Python packages and their versions: pytest 5.4.1, coverage 5.1
• Relevant/affected Python-related VS Code extensions and their versions: vscode-python 2020.5.80290
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Enabled
• Value of the python.languageServer setting: Microsoft

Expected behaviour

Tests are discovered

Actual behaviour

Error message is shown with:

Test discovery error, please check the configuration settings for the tests.

Steps to reproduce:

file: test_simple.py

def test():
    return 43


def test2():
    return 55


def test_thetest():
    assert test() == 43


# def test_thetest2():
#     assert test2() == 55

Running this code with pytest should result in 75% code coverage (4 statements, 3 covered).

1. run: pytest test_simple.py

Logs

FAIL Required test coverage of 100% not reached. Total coverage: 75.00% ============================ no tests ran in 0.65s =============================

Traceback (most recent call last):
File "~/.vscode/extensions/ms-python.python-2020.5.80290/pythonFiles/testing_tools/run_adapter.py", line 22, in
main(tool, cmd, subargs, toolargs)
File "~/.vscode/extensions/ms-python.python-2020.5.80290/pythonFiles/testing_tools/adapter/main.py", line 100, in main
parents, result = run(toolargs, **subargs)
File "~/.vscode/extensions/ms-python.python-2020.5.80290/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 1)

[kimadeline]

Duplicate of #11729, please upvote it to help us prioritize it.

Thank you!

[Brusalk]

@kimadeline I'm not sure that this is actually a duplicate of #11729

I've run into this too now, where I specify the pytest setting --cov-fail-under.

What happens is the pytest execution process fails because code coverage during discovery doesn't yield nearly the same results as when the tests actually run.

============================ no tests ran in 0.93s =============================

Traceback (most recent call last):
  File "~/.vscode-server/extensions/ms-python.python-2020.6.88468/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "~/.vscode-server/extensions/ms-python.python-2020.6.88468/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "~/.vscode-server/extensions/ms-python.python-2020.6.88468/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 1)

When running tests normally, this same package gets 100% code coverage, and the pytest process exits successfully.

[dermotduffy]

Same observation: I don't think this is a duplicate. Adding --cov-fail-under as a pytest argument will (still) cause test discovery to fail in VS Code.

[kimadeline]

Please upvote #6594 and #11729, since they will address the root cause of this issue.

Thank you!

[n4nn31355]

Adding --no-cov in python.testing.pytestArgs setting helps to avoid this behavior. If it's acceptable I can do it in the same way as in #15978

[kimadeline]

Same as #15978 (comment): After discussing it with the team, we do not want to add --no-cov to the default list of arguments as we want to keep our list of defaults short, and let users configure the tools as they need, either by updating the setting globally, or using configuration files for the tools directly.