Coverage Checks

[llchan]

Would it be possible to write a script to measure typeshed coverage for a given package/module, and track that over time? Specifically, I'd like to see the stdlib get to 100% over time. Sometimes stubs exist for a particular module, but they are missing a handful of defs, and we don't know until someone tries to check it in mypy and reports it.

[JelleZijlstra]

We'd definitely like more automated coverage checks. I've started one at https://github.com/JelleZijlstra/stubcheck and mypy recently gained a stub testing script at python/mypy#3036, but we need to do more work to integrate those into typeshed.

[gvanrossum]

Which reminds me, is there any documentation for scripts/stubtest.py? It's got scant docstrings, no usage message except "must provide at least one module to test", and when I invoke it with "fnmatch" (the only thing I've tried so far) I get complaints about modules imported in the runtime version that aren't in the stub for a good reason:

module "os" defined at line 0 at runtime but is not defined in stub
module "re" defined at line 0 at runtime but is not defined in stub
module "functools" defined at line 0 at runtime but is not defined in stub
module "posixpath" defined at line 0 at runtime but is not defined in stub

This also reveals that it doesn't do line numbers right yet, and IMO it should show the full filenames for the things it is comparing.

[JukkaL]

Stubtest still needs some polish. Somebody could spend a little time playing around with it and file issues for simple things that could be improved. I can do this unless somebody else wishes to volunteer. The issues should probably be added to the mypy repo.

[gvanrossum]

OK added python/mypy#3329.

[hauntsaninja]

This issue is similar to #4179. Most of the discussion here centres around stubtest (which has recently received a rewrite, see python/mypy#8325 and others, and is now integrated into typeshed CI), so closing in favour of the newer issue.