typeshed: run stubtest in CI

.travis.yml

@@ -29,3 +29,19 @@ jobs:
     - name: "flake8"
       install: pip install -r requirements-tests-py3.txt
       script: flake8
+    - name: "stubtest py38"
+      python: 3.8
+      install: pip install -U git+git://github.com/python/mypy@a07dbd00
+      script: ./tests/stubtest_test.py
+    - name: "stubtest py37"
+      python: 3.7
+      install: pip install -U git+git://github.com/python/mypy@a07dbd00
+      script: ./tests/stubtest_test.py
+    - name: "stubtest py36"
+      python: 3.6
+      install: pip install -U git+git://github.com/python/mypy@a07dbd00
+      script: ./tests/stubtest_test.py
+    - name: "stubtest py35"
+      python: 3.5
+      install: pip install -U git+git://github.com/python/mypy@a07dbd00
+      script: ./tests/stubtest_test.py

README.md

@@ -86,24 +86,22 @@ requests. If you have questions related to contributing, drop by the [typing Git
 ## Running the tests
 
 The tests are automatically run by Travis CI on every PR and push to
-the repo.  There are several sets of tests: `tests/mypy_test.py`
-runs tests against [mypy](https://github.com/python/mypy/), while
-`tests/pytype_test.py` runs tests against
-[pytype](https://github.com/google/pytype/).
-
-Both sets of tests are shallow -- they verify that all stubs can be
-imported but they don't check whether stubs match their implementation
-(in the Python standard library or a third-party package).  Also note
-that each set of tests has a blacklist of modules that are not tested
-at all.  The blacklists also live in the tests directory.
+the repo. Note that it can be useful to enable Travis CI on your own fork of
+typeshed.
 
-In addition, you can run `tests/mypy_selftest.py` to run mypy's own
-test suite using the typeshed code in your repo. This will sometimes
-catch issues with incorrectly typed stubs, but is much slower than the
-other tests.
+There are several tests:
+- `tests/mypy_test.py`
+runs tests against [mypy](https://github.com/python/mypy/)
+- `tests/pytype_test.py` runs tests against
+[pytype](https://github.com/google/pytype/).
+- `tests/mypy_selftest.py` runs mypy's test suite using this version of
+typeshed.
+- `tests/check_consistent.py` checks certain files in typeshed remain
+consistent with each other.
+- `tests/stubtest_test.py` checks stubs against the objects at runtime.
+- `flake8` enforces a style guide.
 
-To manually run the mypy tests, you need to have Python 3.5 or higher;
-Python 3.6.1 or higher is recommended.
+### Setup
 
 Run:
 ```
@@ -112,31 +110,74 @@ $ source .venv3/bin/activate
 (.venv3)$ pip3 install -r requirements-tests-py3.txt
 ```
 This will install mypy (you need the latest master branch from GitHub),
-typed-ast, flake8, and pytype. You can then run mypy, flake8, and pytype tests
-by invoking:
-```
-(.venv3)$ python3 tests/mypy_test.py
-...
-(.venv3)$ python3 tests/mypy_selftest.py
-...
-(.venv3)$ flake8
-...
-(.venv3)$ python3 tests/pytype_test.py
-...
-```
-Note that flake8 only works with Python 3.6 or higher, and that to run the
-pytype tests, you will need Python 2.7 and Python 3.6 interpreters. Pytype will
-find these automatically if they're in `PATH`, but otherwise you must point to
-them with the `--python27-exe` and `--python36-exe` arguments, respectively.
+typed-ast, flake8 (and plugins), pytype, black and isort.
+
+### mypy_test.py
 
-For mypy, if you are in the typeshed repo that is submodule of the
+This test requires Python 3.5 or higher; Python 3.6.1 or higher is recommended.
+Run using:`(.venv3)$ python3 tests/mypy_test.py`
+
+This test is shallow — it verifies that all stubs can be
+imported but doesn't check whether stubs match their implementation
+(in the Python standard library or a third-party package). It has a blacklist of
+modules that are not tested at all, which also lives in the tests directory.
+
+If you are in the typeshed repo that is submodule of the
 mypy repo (so `..` refers to the mypy repo), there's a shortcut to run
 the mypy tests that avoids installing mypy:
 ```bash
 $ PYTHONPATH=../.. python3 tests/mypy_test.py
 ```
-You can mypy tests to a single version by passing `-p2` or `-p3.5` e.g.
+You can restrict mypy tests to a single version by passing `-p2` or `-p3.5`:
 ```bash
 $ PYTHONPATH=../.. python3 tests/mypy_test.py -p3.5
 running mypy --python-version 3.5 --strict-optional # with 342 files
 ```
+
+### pytype_test.py
+
+This test requires Python 2.7 and Python 3.6. Pytype will
+find these automatically if they're in `PATH`, but otherwise you must point to
+them with the `--python27-exe` and `--python36-exe` arguments, respectively.
+Run using: `(.venv3)$ python3 tests/pytype_test.py`
+
+This test works similarly to `mypy_test.py`, except it uses `pytype`.
+
+### mypy_selftest.py
+
+This test requires Python 3.5 or higher; Python 3.6.1 or higher is recommended.
+Run using: `(.venv3)$ python3 tests/mypy_selftest.py`
+
+This test runs mypy's own test suite using the typeshed code in your repo. This
+will sometimes catch issues with incorrectly typed stubs, but is much slower
+than the other tests.
+
+### check_consistent.py
+
+Run using: `python3 tests/check_consistent.py`
+
+### stubtest_test.py
+
+This test requires Python 3.5 or higher.
+Run using `(.venv3)$ python3 tests/stubtest_test.py`
+
+This test compares the stdlib stubs against the objects at runtime. Because of
+this, the output depends on which version of Python it is run with.
+If you need a specific version of Python to repro a CI failure,
+[pyenv](https://github.com/pyenv/pyenv) can help (as can enabling Travis CI on
+your fork).
+
+Due to its dynamic nature, you may run into false positives. In this case, you
+can add to the whitelists for each affected Python version in
+`tests/stubtest_whitelists`. Please file issues for stubtest false positives
+at [mypy](https://github.com/python/mypy/issues).
+
+To run stubtest against third party stubs, it's easiest to use stubtest
+directly. stubtest can also help you find things missing from the stubs.
+
+
+### flake8
+
+flake8 requires Python 3.6 or higher. Run using: `(.venv3)$ flake8`
+
+Note typeshed uses the `flake8-pyi` and `flake8-bugbear` plugins.

tests/stubtest_test.py

@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+"""Test typeshed using stubtest
+
+stubtest is a script in the mypy project that compares stubs to the actual objects at runtime.
+Note that therefore the output of stubtest depends on which Python version it is run with.
+In typeshed CI, we run stubtest with each Python minor version from 3.5 through 3.8 inclusive.
+
+We pin the version of mypy / stubtest we use in .travis.yml so changes to those don't break
+typeshed CI.
+
+"""
+
+from pathlib import Path
+import subprocess
+import sys
+
+
+def run_stubtest(typeshed_dir: Path) -> int:
+    whitelist_dir = typeshed_dir / "tests" / "stubtest_whitelists"
+    version_whitelist = "py{}{}.txt".format(sys.version_info.major, sys.version_info.minor)
+
+    cmd = [
+        sys.executable,
+        "-m",
+        "mypy.stubtest",
+        # Use --ignore-missing-stub, because if someone makes a correct addition, they'll need to
+        # also make a whitelist change and if someone makes an incorrect addition, they'll run into
+        # false negatives.
+        "--ignore-missing-stub",
+        "--check-typeshed",
+        "--custom-typeshed-dir",
+        str(typeshed_dir),
+        "--whitelist",
+        str(whitelist_dir / "py3_common.txt"),
+        "--whitelist",
+        str(whitelist_dir / version_whitelist),
+    ]
+    if sys.version_info < (3, 8):
+        # As discussed in https://github.com/python/typeshed/issues/3693, we only aim for
+        # positional-only arg accuracy for the latest Python version.
+        cmd += ["--ignore-positional-only"]
+    try:
+        print(" ".join(cmd), file=sys.stderr)
+        subprocess.run(cmd, check=True)
+    except subprocess.CalledProcessError as e:
+        print(
+            "\nNB: stubtest output depends on the Python version it is run with. See README.md "
+            "for more details.\n"
+            "NB: We only check positional-only arg accuracy for Python 3.8.\n"
+            "If stubtest is complaining about 'unused whitelist entry' after your fix, please "
+            "remove the entry from the whitelist file. Note you may have to do this for other "
+            "version-specific whitelists as well. Thanks for helping burn the backlog of errors!\n"
+            "\nCommand run was: {}\n".format(" ".join(cmd)),
+            file=sys.stderr,
+        )
+        print("stubtest failed", file=sys.stderr)
+        return e.returncode
+    else:
+        print("stubtest succeeded", file=sys.stderr)
+        return 0
+
+
+if __name__ == "__main__":
+    sys.exit(run_stubtest(typeshed_dir=Path(".")))

tests/stubtest_whitelists/py35.txt

@@ -0,0 +1,106 @@
+asyncio.Future._callbacks
+asyncio.Future._exception
+asyncio.Future._loop
+asyncio.Future._tb_logger
+asyncio.Task.__init__
+asyncio.constants.DEBUG_STACK_DEPTH
+asyncio.constants.SENDFILE_FALLBACK_READBUFFER_SIZE
+asyncio.constants.SSL_HANDSHAKE_TIMEOUT
+asyncio.exceptions
+asyncio.futures.Future._callbacks
+asyncio.futures.Future._exception
+asyncio.futures.Future._loop
+asyncio.futures.Future._tb_logger
+asyncio.futures._TracebackLogger.__init__
+asyncio.protocols.BufferedProtocol
+asyncio.runners
+asyncio.tasks.Task.__init__
+bdb.GENERATOR_AND_COROUTINE_FLAGS
+builtins.str.maketrans
+cgi.escape
+cmath.log
+codecs.StreamRecoder.seek
+collections.Reversible
+collections.UserString.maketrans
+ctypes.CDLL.__init__
+decimal.Decimal.as_integer_ratio
+gettext.NullTranslations.npgettext
+gettext.NullTranslations.pgettext
+gettext.dnpgettext
+gettext.dpgettext
+gettext.npgettext
+gettext.pgettext
+importlib.metadata
+importlib.resources
+io.BufferedRandom.read1
+io.BufferedReader.read1
+io.StringIO.readline
+ipaddress._BaseNetwork.__init__
+macpath.basename
+macpath.commonpath
+macpath.commonprefix
+macpath.dirname
+macpath.getatime
+macpath.getctime
+macpath.getmtime
+macpath.getsize
+macpath.isabs
+macpath.isdir
+macpath.islink
+macpath.ismount
+macpath.join
+macpath.normpath
+macpath.realpath
+macpath.relpath
+macpath.samefile
+macpath.samestat
+macpath.split
+macpath.splitdrive
+macpath.splitext
+mmap.ACCESS_DEFAULT
+modulefinder.ModuleFinder.scan_opcodes
+multiprocessing.context.BaseContext.reducer
+multiprocessing.shared_memory
+os.DirEntry
+os.utime
+pyexpat.XMLParserType.ExternalEntityParserCreate
+smtpd.SMTPChannel.__init__
+smtpd.SMTPServer.__init__
+socket.NETLINK_CRYPTO
+sre_compile.dis
+sre_parse.GLOBAL_FLAGS
+sre_parse.Tokenizer.pos
+sre_parse.Verbose
+subprocess.check_output
+tracemalloc.Filter.__init__
+typing.AbstractSet.isdisjoint
+typing.Coroutine.cr_await
+typing.Coroutine.cr_code
+typing.Coroutine.cr_frame
+typing.Coroutine.cr_running
+typing.Generator.gi_code
+typing.Generator.gi_frame
+typing.Generator.gi_running
+typing.Generator.gi_yieldfrom
+typing.Mapping.get
+typing.MutableMapping.pop
+typing.MutableMapping.setdefault
+typing.MutableSequence.append
+typing.MutableSequence.extend
+typing.MutableSequence.insert
+typing.MutableSequence.remove
+typing.MutableSet.add
+typing.MutableSet.discard
+typing.MutableSet.remove
+typing.Protocol
+typing.Sequence.count
+typing.Sequence.index
+typing.Type
+typing.runtime_checkable
+unittest.async_case
+uuid.UUID.int
+uuid.getnode
+xml.etree.cElementTree.XMLPullParser
+xml.parsers.expat.XMLParserType.ExternalEntityParserCreate
+zipfile.ZipFile.open
+zlib.compress