Typing Stubs and PEP 561 compatibility

[simonjayhawkins]

xref #28135 (comment)

do we want to make pandas PEP 561 compatible?

https://mypy.readthedocs.io/en/latest/installed_packages.html#making-pep-561-compatible-packages

[jbrockmendel]

If we go down that road, we could consider using stub files in cython to make that code valid python (which would allow us to lint)

[zero323]

@WillAyd @jbrockmendel @simonjayhawkins How would you describe current state of things? I've seen that many core components already have pretty good API coverage. Is it realistic to expect this to happen in foreseeable future (let's say next release or two)?

To explain the context of this question ‒ for the last three years I've been working on stub files for Apache Spark. Over this time contact surface between Pandas and PySpark grown significantly, mostly due to introduction and active development of so-called Pandas udfs. Since Pandas doesn't advertise its annotation it effectively creates a growing gap, in practice not covered by type checkers.
Additionally the latest upstream developments utilize type hints in Pandas-dependent components, leading conflicts between static type checking, and upstream runtime requirements.

Furthermore lack of actionable annotations leads to rather ugly escalations in case of polymorphic functions, which accept Pandas objects, as well as other types,

Now... For some time, to partially address the problem, I've been using Protocols and dummy compatibility imports. The idea is basically to:

• Extract existing annotations from Pandas (example)
• Type ignore missing Pandas imports
• Provide intermediate annotations.

This approach is not without its own problems, but does the trick. If Pandas is going to PEP 561 these will become obsolete, but if such move is not going to happen any time soon, I will consider formalizing this approach, and agitating for required adjustments in core PySpark. However, given amount of red tape, I'd really like to avoid it :)

[WillAyd]

@zero323 are you asking if we plan on distributing stub files? If so I would say no for now; we have been annotating inline. Not fully complete but you'll see issues like #26766 used to track that progress

[zero323]

Thank you for the answer @WillAyd.

I didn't mean stubs explicitly, but annotations that are visible to external checkers (which tend to assume no annotations, if no stubs packages or py.typed files are present).

For me it is basically a decision between waiting a bit longer (if these are expected to be ready soon. From that perspective even dynamic annotations are good enough) or adding elaborate workarounds.

[mfcabrera]

As an end user I am also having the use of annotationg code with say pd.DataFrame and mypy interpreting it as Any.

I am bit confused here. in order to distribute the annotations isn't that enough to add py.typed file and modify the setup.py? That way mypy will be able to use the inline annotated code right? or there is something I am missing?

Update:
Sorry I did not see #28831

However I am bit still confused on what is need to make this happen (if it is happening).

[zero323]

I am bit confused here. in order to distribute the annotations isn't that enough to add py.typed file and modify the setup.py?

There is a bit more to that. In general it is good to have annotations that pass type checks with standardish settings, otherwise you're likely to break things downstream.

[OliverSieweke]

Related issue: #12412