When a test case name is prefix of another the wrong tests are run

[memeplex]

Environment data

• VS Code version: 1.38.1
• Extension version: 2019.9.34911
• OS and version: Ubuntu 19.04
• Python version: 3.7.3
• Type of virtual environment used: N/A
• Relevant/affected Python packages and their versions: unittest
• Jedi or Language Server?: Jedi

Expected behaviour

Running tests by clicking the name of a TestCase in the test view will run the tests in that TestCase.

Actual behaviour

When the names of two TestCases share the same prefix, only the tests for the one with the longer name are run, as shown in the following screenshot:

Steps to reproduce:

1. Create a folder with contents:

<folder>/
     tests/
           __init__.py
           test_stuff.py

2. Copy the code in the screenshot above into test_stuff.py.
3. Open the test view as in the screenshot above.
4. Double click on TestStuff.
5. Double click on TestStuffBuilder.
6. Only test_b is run.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

User belongs to experiment group 'ShowExtensionSurveyPrompt - control'
Starting Jedi Python language engine.
> conda --version
> pyenv root
> python3.7 -c "import sys;print(sys.executable)"
> python3.6 -c "import sys;print(sys.executable)"
> python3 -c "import sys;print(sys.executable)"
> python2 -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> /bin/python3 -c "import sys;print(sys.executable)"
> conda info --json
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3 -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover(".", pattern="test_*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: /tmp/test
> /bin/python3 -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover(".", pattern="test_*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: /tmp/test
> conda info --json
> conda --version
> /bin/python3 -c "import sys;print(sys.prefix)"
cwd: /tmp/test
> /bin/python3 -c "import sys;print(sys.prefix)"
cwd: /tmp/test
> /bin/python3 -c "import sys;print(sys.executable)"
cwd: /tmp/test
> /bin/python3 -c "import sys;print(sys.executable)"
cwd: /tmp/test
> /bin/python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
cwd: /tmp/test
> /bin/python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
cwd: /tmp/test
> /bin/python3 -m site --user-site
cwd: /tmp/test
> /bin/python3 -m site --user-site
cwd: /tmp/test
> /bin/python3 completion.py
cwd: ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles
> /bin/python3 completion.py
cwd: ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/test_stuff.py
cwd: /tmp/test
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/test_stuff.py
cwd: /tmp/test
##########Linting Output - flake8##########
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/__init__.py
cwd: /tmp/test
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/__init__.py
cwd: /tmp/test
##########Linting Output - flake8##########
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/__init__.py
cwd: /tmp/test
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/__init__.py
cwd: /tmp/test
##########Linting Output - flake8##########
> /bin/python3 -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover(".", pattern="test_*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: /tmp/test
> /bin/python3 -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover(".", pattern="test_*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=45175 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=45175 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=39933 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=39933 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=43647 -ttests.TestStuff.test_a --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=43647 -ttests.TestStuff.test_a --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=33905 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=33905 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=37247 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=37247 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=39113 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=39113 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=33985 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=33985 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=46605 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=46605 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=41825 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=41825 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test

Test Logs

start
start
tests.TestStuff.test_a
tests.TestStuffBuilder.test_b
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_a (tests.TestStuff) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

[karrtikr]

I am able to confirm this, thanks! I assume you're using unittest framework.

Double click on TestStuff.

I am confused here😕 AFAICS double clicking a item in the test explorer does nothing but opens the file. Are you referring to the play icon next to each test item? I used that to run the TestCase.

[memeplex]

I am confused hereconfused AFAICS double clicking a item in the test explorer

You're right, sorry, I should have said "clicking on the arrow/play icon".

I assume you're using unittest framework.

Yes, just the builtin unittest framework.

[memeplex]

@karrtikr you added the "needs PR" label, does it mean that you're expecting a PR from me or is it more like an internal label?