pytest.warns and pytest.raises ordering issue

[jrbourbeau]

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

Description

It appears that when testing that both a warning is emitted and an exception is raised, the order in which pytest.raises and pytest.warns are used matters and can lead to potentially confusing behavior.

Minimal example

For example, the following test passes:

import warnings

import pytest

def my_func():
    # This both emits a `FutureWarning` and raises a `ValueError`
    warnings.warn("my warning", category=FutureWarning)
    raise ValueError("my error")

def test():
    with pytest.raises(ValueError, match="my error"):
        with pytest.warns(SyntaxWarning, match="my warning"): # <-- I expected this to fail because `my_func` doesn't emit a `SyntaxWarning`
            my_func()

Naively I expected this test to fail as my_func is emitting a FutureWarning, but the test checking that a SyntaxWarning is emitted.

However, if I switch the order of pytest.raises and pytest.warns, then the test fails as expected. That is

import warnings

import pytest

def my_func():
    # This both emits a `FutureWarning` and raises a `ValueError`
    warnings.warn("my warning", category=FutureWarning)
    raise ValueError("my error")

def test():
    with pytest.warns(SyntaxWarning, match="my warning"):
        with pytest.raises(ValueError, match="my error"):
            my_func()

outputs

__________________________________________________________________________ test __________________________________________________________________________

    def test():
        with pytest.warns(SyntaxWarning, match="my warning"):
            with pytest.raises(ValueError, match="my error"):
>               my_func()
E               Failed: DID NOT WARN. No warnings of type (<class 'SyntaxWarning'>,) was emitted. The list of emitted warnings is: [FutureWarning('my warning')].

test.py:13: Failed

Extra info

pip list:

Package                       Version                    Location
----------------------------- -------------------------- --------------------------------------
aiobotocore                   1.3.3
aiohttp                       3.7.4.post0
aioitertools                  0.7.1
alabaster                     0.7.12
anyio                         3.3.0
appdirs                       1.4.4
appnope                       0.1.2
argon2-cffi                   20.1.0
async-generator               1.10
async-timeout                 3.0.1
asyncssh                      2.7.0
attrs                         21.2.0
awscli                        1.19.106
Babel                         2.9.1
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
bcrypt                        3.2.0
bleach                        3.3.1
blosc                         1.10.2
bokeh                         2.3.3
botocore                      1.20.106
brotlipy                      0.7.0
cached-property               1.5.2
certifi                       2021.5.30
cffi                          1.14.6
cfgv                          3.3.0
cftime                        1.5.0
chardet                       4.0.0
charset-normalizer            2.0.0
click                         8.0.1
cloudpickle                   1.6.0
cmarkgfm                      0.6.0
colorama                      0.4.3
cryptography                  3.4.7
cytoolz                       0.11.0
dask                          2021.8.1+3.ga6cc44fd.dirty /Users/james/projects/dask/dask
dask-labextension             5.1.0
dask-sphinx-theme             1.3.6
debugpy                       1.4.1
decorator                     5.0.9
defusedxml                    0.7.1
distlib                       0.3.2
distributed                   2021.8.1                   /Users/james/projects/dask/distributed
docutils                      0.16
editdistance-s                1.0.0
entrypoints                   0.3
execnet                       1.9.0
filelock                      3.0.12
fsspec                        2021.7.0+8.gee22435
future                        0.18.2
gssapi                        1.6.14
h5py                          3.3.0
HeapDict                      1.0.1
identify                      2.2.11
idna                          3.1
imagesize                     1.2.0
importlib-metadata            4.6.1
iniconfig                     1.1.1
ipykernel                     6.0.3
ipython                       7.25.0
ipython-genutils              0.2.0
ipywidgets                    7.6.3
jedi                          0.18.0
Jinja2                        3.0.1
jmespath                      0.10.0
joblib                        1.1.0.dev0
json5                         0.9.5
jsonschema                    3.2.0
jupyter-client                6.1.12
jupyter-core                  4.7.1
jupyter-server                1.10.1
jupyter-server-proxy          3.1.0
jupyterlab                    3.1.1
jupyterlab-pygments           0.1.2
jupyterlab-server             2.6.1
jupyterlab-widgets            1.0.0
Keras                         2.4.3
keyring                       23.0.1
locket                        0.2.0
lz4                           3.1.3
MarkupSafe                    2.0.1
matplotlib-inline             0.1.2
mistune                       0.8.4
more-itertools                8.8.0
msgpack                       1.0.2
multidict                     5.1.0
nbclassic                     0.3.1
nbclient                      0.5.3
nbconvert                     6.1.0
nbformat                      5.1.3
nest-asyncio                  1.5.1
netCDF4                       1.5.7
nodeenv                       1.6.0
notebook                      6.4.0
numpy                         1.21.1
numpydoc                      1.1.0
olefile                       0.46
packaging                     21.0
pandas                        1.3.1
pandocfilters                 1.4.2
paramiko                      2.7.2
parso                         0.8.2
partd                         1.2.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        8.2.0
pip                           21.2.1
pkginfo                       1.7.1
pluggy                        0.13.1
pre-commit                    2.13.0
prometheus-client             0.11.0
prompt-toolkit                3.0.19
psutil                        5.8.0
ptyprocess                    0.7.0
py                            1.10.0
pyasn1                        0.4.8
pycparser                     2.20
Pygments                      2.9.0
PyNaCl                        1.4.0
pynvml                        11.0.0
pyOpenSSL                     20.0.1
pyparsing                     2.4.7
pyrsistent                    0.17.3
PySocks                       1.7.1
pytest                        6.2.4
pytest-asyncio                0.12.0
pytest-faulthandler           2.0.1
pytest-forked                 1.3.0
pytest-repeat                 0.8.0
pytest-rerunfailures          10.1
pytest-timeout                1.4.2
pytest-xdist                  2.3.0
python-dateutil               2.8.2
python-snappy                 0.6.0
pytz                          2021.1
PyYAML                        5.4.1
pyzmq                         22.1.0
readme-renderer               27.0
requests                      2.26.0
requests-toolbelt             0.9.1
requests-unixsocket           0.2.0
rfc3986                       1.5.0
rsa                           4.7.2
s3fs                          2021.7.0+5.g0df4f28
s3transfer                    0.4.2
scikit-learn                  0.24.2
scipy                         1.7.0
Send2Trash                    1.7.1
setuptools                    49.6.0.post20210108
simpervisor                   0.4
six                           1.16.0
sniffio                       1.2.0
snowballstemmer               2.1.0
sortedcollections             2.1.0
sortedcontainers              2.4.0
Sphinx                        4.1.2
sphinx-click                  3.0.1
sphinx-rtd-theme              0.5.2
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
tblib                         1.7.0
terminado                     0.10.1
testpath                      0.5.0
threadpoolctl                 2.2.0
toml                          0.10.2
toolz                         0.11.1
torch                         1.9.0
torchvision                   0.9.0a0
tornado                       6.1
tqdm                          4.61.2
traitlets                     5.0.5
twine                         0.0.0
typing-extensions             3.10.0.0
urllib3                       1.26.6
virtualenv                    20.4.7
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              0.57.0
wheel                         0.36.2
widgetsnbextension            3.5.1
wrapt                         1.12.1
yarl                          1.6.3
zict                          2.0.0
zipp                          3.5.0
zstandard                     0.15.2

I'm using pytest==6.2.4 on a Mac running Big Sur.