3.10 objects.inv classifies many types as data

[gaborbernat]

BPO	45391
Nosy	@ambv, @bskinn, @miss-islington, @gaborbernat, @Fidget-Spinner
PRs	bpo-45391: mark UnionType as a class in documentation #28757 [3.10] bpo-45391: mark UnionType as a class in documentation (GH-28757) #30004

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-06.12:20:50.089>
labels = ['3.10', 'docs']
title = '3.10 objects.inv classifies many types as data'
updated_at = <Date 2021-12-09.13:22:02.376>
user = 'https://github.com/gaborbernat'

bugs.python.org fields:

activity = <Date 2021-12-09.13:22:02.376>
actor = 'kj'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2021-10-06.12:20:50.089>
creator = 'gaborjbernat'
dependencies = []
files = []
hgrepos = []
issue_num = 45391
keywords = ['patch']
message_count = 11.0
messages = ['403300', '403301', '403308', '403310', '403311', '403313', '403318', '403343', '408115', '408119', '408120']
nosy_count = 7.0
nosy_names = ['docs@python', 'lukasz.langa', 'bskinn', 'miss-islington', 'gaborjbernat', 'dom1310df', 'kj']
pr_nums = ['28757', '30004']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue45391'
versions = ['Python 3.10']

[gaborbernat]

It's a class though:

❯ sphobjinv suggest ./objects.inv UnionType
:py:data:`types.UnionType`

defined as:

UnionType = type(int | str)

[gaborbernat]

The issue with the current state this is that intersphinx fails to find types.UnionType in objects.inv because of leaves under the incorrect namespace (data vs class).

[ambv]

On the PR you mention there's more types with this problem. Can we get a full list?

[gaborbernat]

Not easily, but, e.g. the EllipsisType is one. I would have to create some script which I haven't done yet.

The best would be to create a sphinx plugin that collects entries registered in the doc and displays the discrepancy against the intershpinx object. This way, we could defend against future such issues too. I can give it a go in a few days if no one else wants to do so before that.

[bskinn]

If I understand the problem correctly, these mis-attributions of roles (to 'data' instead of 'class' come about when a thing that is technically a class is defined in source using simple assignment, as with UnionType.

Problematic entries will thus have 'data' as role, and their identifiers will be camel-cased.

So, as a quick search to identify likely candidates:

>> import re, sphobjinv as soi
>> from pprint import pprint
>> inv = soi.Inventory(url="https://docs.python.org/3.10/objects.inv")

# Find entries where the first character after the final period
# is uppercase, and the second character after the final period
# is lowercase.
>>> pat = re.compile(r"([.][A-Z][a-z])[^.]*$")

>> pprint([obj.name for obj in inv.objects if obj.role == "data" and pat.search(obj.name)])

['_thread.LockType',
'ast.PyCF_ALLOW_TOP_LEVEL_AWAIT',
'ast.PyCF_ONLY_AST',
'ast.PyCF_TYPE_COMMENTS',
'importlib.resources.Package',
'importlib.resources.Resource',
'socket.SocketType',
'types.AsyncGeneratorType',
'types.BuiltinFunctionType',
'types.BuiltinMethodType',
'types.CellType',
'types.ClassMethodDescriptorType',
'types.CoroutineType',
'types.EllipsisType',
'types.FrameType',
'types.FunctionType',
'types.GeneratorType',
'types.GetSetDescriptorType',
'types.LambdaType',
'types.MemberDescriptorType',
'types.MethodDescriptorType',
'types.MethodType',
'types.MethodWrapperType',
'types.NoneType',
'types.NotImplementedType',
'types.UnionType',
'types.WrapperDescriptorType',
'typing.Annotated',
'typing.Any',
'typing.AnyStr',
'typing.Callable',
'typing.ClassVar',
'typing.Concatenate',
'typing.Final',
'typing.Literal',
'typing.NoReturn',
'typing.Optional',
'typing.ParamSpecArgs',
'typing.ParamSpecKwargs',
'typing.Tuple',
'typing.TypeAlias',
'typing.TypeGuard',
'typing.Union',
'weakref.CallableProxyType',
'weakref.ProxyType',
'weakref.ProxyTypes',
'weakref.ReferenceType']

I would guess those 'ast.PyCF...' objects can be ignored, they appear to be constants?

[bskinn]

Identifiers starting with two uppercase letters returns a HUGE list.

>> pat2 = re.compile(r"([.][A-Z][A-Z])[^.]*$")

Filtering down by only those that contain.lower() "type":

>> pprint([obj.name for obj in inv.objects if obj.role == "data" and pat2.search(obj.name) and "type" in obj.name.lower()])

['errno.EPROTOTYPE',
'locale.LC_CTYPE',
'sqlite3.PARSE_DECLTYPES',
'ssl.CHANNEL_BINDING_TYPES',
'token.TYPE_COMMENT',
'token.TYPE_IGNORE',
'typing.TYPE_CHECKING',
'xml.parsers.expat.XMLParserType']

Of these, only 'xml.parsers.expat.XMLParserType' seems to me a likely problem entry.

[gaborbernat]

I think Brian Skinn script is a rough approximation, but definitely entirely accurate. You'd have to match up what sphinx thinks per doc vs what you import for an accurate view.

[gaborbernat]

Here's a gist where I managed to detect roughly 140 errors (some looks like potential false positive, so likely the real number is more around 100):

https://gist.github.com/gaborbernat/5360badab2125b3f81a3bcbce0e94c2a#file-found_issues-output-L1

This does make a few concessions:

• ignores the difference between function and method; way to many functions are documented as methods and vice-versa to disallow this (or would be a major overhaul)
• https://docs.python.org/3/c-api/structures.html?highlight=meth_class#METH_VARARGS is documented under python domain but IMHO should be C
• https://docs.python.org/3/c-api/typeobj.html?highlight=py_tpflags_base_exc_subclass#c.PyTypeObject.tp_flags is documented under python domain but IMHO should be C
• does not clarifies where to type classes goes - they seem to be a weird in-between a method and a class, satisfying neither - see related discussion on topic from https://bugs.python.org/issue41973

[Fidget-Spinner]

New changeset e2cfc89 by Bernát Gábor in branch 'main':
bpo-45391: mark UnionType as a class in documentation (GH-28757)
e2cfc89

[miss-islington]

New changeset 2c2ee83 by Miss Islington (bot) in branch '3.10':
bpo-45391: mark UnionType as a class in documentation (GH-28757)
2c2ee83

[Fidget-Spinner]

As a start, I merged the types.UnionType fix because it's straightforward, but the rest are a little dubious so I'll leave this issue open for now.

[encukou]

• https://docs.python.org/3/c-api/structures.html?highlight=meth_class#METH_VARARGS is documented under python domain but IMHO should be C
• https://docs.python.org/3/c-api/typeobj.html?highlight=py_tpflags_base_exc_subclass#c.PyTypeObject.tp_flags is documented under python domain but IMHO should be C

I opened #96996 for these.