Skip to content

Improve docs on running stubtest locally #8822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 8, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 34 additions & 13 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,28 +88,27 @@ Run using
This test compares the stdlib stubs against the objects at runtime. Because of
this, the output depends on which version of Python and on what kind of system
it is run.
Thus the easiest way to run this test is via Github Actions on your fork;
if you run it locally, it'll likely complain about system-specific
differences (in e.g, `socket`) that the type system cannot capture.
As such, if you run this test locally, it may complain about system-specific
differences (in e.g, `socket`) that the type system cannot capture or our stubtest settings
in CI do not account for. If you run into this issue, consider opening a draft PR and letting CI
test it automatically (or
[running the test via Github Actions](https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow#running-a-workflow)
on your typeshed fork).

If you need a specific version of Python to repro a CI failure,
[pyenv](https://github.com/pyenv/pyenv) can help.
[pyenv](https://github.com/pyenv/pyenv) can also help.

Due to its dynamic nature, you may run into false positives. In this case, you
can add to the allowlists for each affected Python version in
`tests/stubtest_allowlists`. Please file issues for stubtest false positives
at [mypy](https://github.com/python/mypy/issues).

To run stubtest against third party stubs, it's easiest to use stubtest
directly, with
```
(.venv3)$ python3 -m mypy.stubtest \
--custom-typeshed-dir <path-to-typeshed> \
<third-party-module>
```
stubtest can also help you find things missing from the stubs.

## stubtest\_third\_party.py

:warning: This script downloads and executes arbitrary code from PyPI. Only run
this script locally if you know you can trust the packages you're running
stubtest on.

Run using
```
(.venv3)$ python3 tests/stubtest_third_party.py
Expand All @@ -127,6 +126,28 @@ For each distribution, stubtest ignores definitions listed in a `@tests/stubtest
relative to the distribution. Additional packages that are needed to run stubtest for a
distribution can be added to `@tests/requirements-stubtest.txt`.

### Using stubtest to find objects missing from the stubs

By default, stubtest emits an error if a public object is present at runtime
but missing from the stub. However, this behaviour can be disabled using the
`--ignore-missing-stub` option.

Many third-party stubs packages in typeshed are currently incomplete, and so by
default, `stubtest_third_party.py` runs stubtest with the
`--ignore-missing-stub` option to test our third-party stubs. However, this
option is not used if the distribution has `ignore_missing_stub = false` in the
`tool.stubtest` section of its `tests/METADATA.toml` file. This setting
indicates that the package is considered "complete", for example:
https://github.com/python/typeshed/blob/6950c3237065e6e2a9b64810765fec716252d52a/stubs/emoji/METADATA.toml#L3-L4

You can help make typeshed's stubs more complete by adding
`ignore_missing_stub = false` to the `tests/METADATA.toml` file for a
third-party stubs distribution, running stubtest, and then adding things that
stubtest reports to be missing to the stub. However, note that not *everything*
that stubtest reports to be missing should necessarily be added to the stub.
For some implementation details, it is often better to add allowlist entries
for missing objects rather than trying to match the runtime in every detail.

## typecheck\_typeshed.py

Run using
Expand Down