pythonTestAdapter fails if sockets are disabled for tests

[johnchristopherjones]

Type: Bug

Behaviour

Expected vs. Actual

Actual

When using pytest with pytest_socket to disable network connections in test, the test suite fails. Test Results displays the following error:

vscode_pytest.VSCodePytestError: Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.connect() with host "localhost" (allowed: "localhost").

Meanwhile, the test suite passes with on the terminal with:

pytest --disable-socket

The test suite can be made to pass by opting out of the pythonTestAdapter experiment in Settings:

"python.experiments.optOutFrom": ["pythonTestAdapter"],

Expected

The test runner succeeds like the CLI.

Steps to reproduce:

I've created a minimal reproduction repo:
https://github.com/johnchristopherjones/vscode-python-test-adapter-socket

The really short version is:

1. Install pytest_socket
2. Decorate a test with @pytest.mark.disable_socket:

@pytest.mark.disable_socket
def test_example():
    assert True

3. Try to run the test with the test runner. (⌘; C)

Or, alternatively:

1. Install pytest_socket

2. Add this to your pyproject.toml:

   [tool.pytest.ini_options]
   addopts = [
       "--disable-socket",    # Disable network in test with pytest-socket

3. Try to run any tests. (⌘; A)

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.6
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Pylance

Output for Python in the Python Test Log panel

Starting now, all test run output will be sent to the Test Result panel, while test discovery output will be sent to the "Python" output channel instead of the "Python Test Log" channel. The "Python Test Log" channel will be deprecated within the next month. See https://github.com/microsoft/vscode-python/wiki/New-Method-for-Output-Handling-in-Python-Testing for details.CLIENT: Server listening on port 50821...
Received JSON data in run script
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /Users/jcj/src/vscode-python-test-adapter-socket
configfile: pyproject.toml
plugins: socket-0.6.0
collected 1 item

tests/test_always_fail.py Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
If you are on a Windows machine, this error may be occurring if any of your tests clear environment variables as they are required to communicate with the extension. Please reference https://docs.pytest.org/en/stable/how-to/monkeypatch.html#monkeypatching-environment-variablesfor the correct way to clear environment variables during testing.


INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 721, in send_post_request
INTERNALERROR>     __socket.connect()
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/testing_tools/socket_manager.py", line 32, in connect
INTERNALERROR>     self.socket = socket.socket(
INTERNALERROR>                   ^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pytest_socket.py", line 80, in __new__
INTERNALERROR>     raise SocketBlockedError()
INTERNALERROR> pytest_socket.SocketBlockedError: A test tried to use socket.socket.
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 226, in call_and_report
INTERNALERROR>     hook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/terminal.py", line 574, in pytest_runtest_logreport
INTERNALERROR>     *self.config.hook.pytest_report_teststatus(report=rep, config=self.config)
INTERNALERROR>      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 62, in _multicall
INTERNALERROR>     next(wrapper_gen)  # first yield
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 233, in pytest_report_teststatus
INTERNALERROR>     execution_post(
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 660, in execution_post
INTERNALERROR>     send_post_request(payload)
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 732, in send_post_request
INTERNALERROR>     raise VSCodePytestError(error_msg)
INTERNALERROR> vscode_pytest.VSCodePytestError: Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
If you are on a Windows machine, this error may be occurring if any of your tests clear environment variables as they are required to communicate with the extension. Please reference https://docs.pytest.org/en/stable/how-to/monkeypatch.html#monkeypatching-environment-variablesfor the correct way to clear environment variables during testing.

Traceback (most recent call last):
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 721, in send_post_request
    __socket.connect()
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/testing_tools/socket_manager.py", line 32, in connect
    self.socket = socket.socket(
                  ^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pytest_socket.py", line 80, in __new__
    raise SocketBlockedError()
pytest_socket.SocketBlockedError: A test tried to use socket.socket.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/run_pytest_script.py", line 67, in <module>
    pytest.main(arg_array)
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
    return wrap_session(config, _main)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 306, in wrap_session
    config.hook.pytest_sessionfinish(
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 130, in _multicall
    teardown[0].send(outcome)
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/terminal.py", line 857, in pytest_sessionfinish
    outcome.get_result()
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 367, in pytest_sessionfinish
    execution_post(
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 660, in execution_post
    send_post_request(payload)
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 732, in send_post_request
    raise VSCodePytestError(error_msg)
vscode_pytest.VSCodePytestError: Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
Starting now, all test run output will be sent to the Test Result panel, while test discovery output will be sent to the "Python" output channel instead of the "Python Test Log" channel. The "Python Test Log" channel will be deprecated within the next month. See https://github.com/microsoft/vscode-python/wiki/New-Method-for-Output-Handling-in-Python-Testing for details.

User Settings

languageServer: "Pylance"

testing
• pytestEnabled: true

terminal
• launchArgs: "<placeholder>"

Extension version: 2023.19.13031019
VS Code version: Code 1.83.1 (f1b07bd25dfad64b0167beb15359ae573aecd2cc, 2023-10-10T23:57:32.750Z)
OS version: Darwin arm64 22.6.0
Modes:

[eleanorjboyd]

Hello! What is the reason you want to disable sockets? The sockets are how we send information on the run status back to the Python extension and gather test results. Therefore it would require some design changes to fix this.

[karthiknadig]

@eleanorjboyd It looks like https://pypi.org/project/pytest-socket/ plugin specifically used for disabling sockets for testing.

[johnchristopherjones]

@eleanorjboyd As @karthiknadig mentioned, sockets are disabled by the pytest-socket plugin for pytest, which is a good practice to ensure the code under test is not opening unexpected network connections. For instance, when testing code that might exercise a third-party API, you want to make sure your test suite isn't REALLY sending requests to that API—especially if your test suite runs in CI.

As I mention in my reproduction repo, I suspect this might have a straightforward cause and possibly a straightforward fix. The vscode_pytest/run_pytest_script.py script is directly invoking pytest.run() in-process. That means the script itself is being subjected to the test suite, which can include all kinds of modifications to the runtime that are very reasonable for test suites that are not at all reasonable for production code, such as patching os.environ and very routine test activities such as opening and closing any number of sockets and files.

I think the test adapter script shouldn't run in the same process as the test suite. Consider spinning off a subprocess, perhaps with asyncio.subprocess.

[johnchristopherjones]

Just for reference, here are some of the Python testing libraries that patch socket.socket:

Package	Oct 2023 Downloads	Testing	Pytest Plugin	Useful in Prod?
pytest-socket	588k	✅	✅	❌
httpretty	487k	✅	❌	✅
pytest-recording	144k	✅	✅	❌
pytest-network	13k	✅	✅	❌

It's also relatively common to just patch socket.socket yourself. For extra convenience and safety, the standard library comes with unittest.mock. That is, a library isn't really necessary, so library usage is an incomplete measure of the impact.

[eleanorjboyd]

Hello! Thank you for the information as it provides important data about this problem. Would named pipes be a sufficient work-around here or do you anticipate that not working for the libraries you listed above?

[johnchristopherjones]

@eleanorjboyd As in os.mkfifo? I'm not very experienced with it. With some quick testing, it does appear to work.

N.B.: The docs effective say it requires a POSIX system. Specifically:

Availability: Unix, not Emscripten, not WASI.

No idea if that affects non-WSL Windows or vscode.dev.