Let plugins customize class MRO

[snarkmaster]

The Python metaclass data model allows the mro() method to be overloaded. This is a genuinely useful feature for dealing with metaclasses whose classes must support inheritance. See "Example" below.

I wrote a mypy plugin for type-checking instances of my metaclass. I was able to model its semantics almost correctly with the existing plugin hooks, except for one thing — customizing the MRO.

The best way I can get my type-checker unblocked is by adding a plugin hook for customizing the MRO. My idea is roughly this:

• Move calculate_class_mro to be a member of SemanticAnalyzerPass2
• Add a plugin hook that gets to customize the computed MRO.

@JukkaL, does this sound like a feature you would take?

Example of needing to customize the MRO:

• My metaclass M injects a implementation-detail base M_classname_base into each class it creates. M's semantics demand that M_classname_base be at the very end of the MRO (to allow overloading of its features).
• Let us define class A(metaclass=M) and class B(A).
• Each class injects its own base: M_A_base and M_B_base. That means that B actually has both implementation-detail bases.
• Unfortunately, M_A_base and M_B_base conflict, and M_B_base ends up after M_A_base in the MRO, which is the opposite of what you want (child must overload parent, not vice versa).
• At runtime, M.mro() can simply remove the A-produced base from B's MRO, and all is well.
• In current mypy, removing M_A_base from B's MRO seems impossible.

Cc: @carljm, if you're curious.

[JukkaL]

At first this sounds like a pretty simple change, but there are some edge cases:

• What if modifying the MRO changes the abstract status of the class?
• What if modifying the MRO turns the class into a named tuple (or a tuple base class)? Or makes it a non-tuple?
• What if metaclass structure becomes inconsistent?

There probably are more things that could go wrong. Maybe you could go through all TypeInfo attributes and analyze which of them might be stale after an MRO change? Maybe we should reject certain MRO changes if they are hard to support?

It would also help if we can identify other projects that would benefit from this (at at least single well-known project).

[snarkmaster]

Here's a proof-of-concept patch: snarkmaster@63f4aa4

I think your concerns are totally reasonable. Changing the MRO is "very advanced use-case" territory, and you can really break the semantics of your classes.

However, let's assume for a moment that the author of the metaclass knows what they are doing — if not, their runtime ends up utterly broken anyway. As I see it, the job of mypy is to allow statically modeling runtime semantics using the parsed AST.

At the moment, all the other plugin hooks are 'at your own risk' — I can completely rewrite a class or its bases from the metaclass or base class hook.

As such, I think it's not too bad to give plugin authors a very simple hook to emulate a well-established part of Python's data model. See the docs here:

https://docs.python.org/3/library/stdtypes.html?highlight=mro#class.mro

What do you think about the simple patch?

[snarkmaster]

PS You will notice in the patch that I deviated from the pattern of "some_hook(fullname) returns a callback, which takes a context". I did this as a suggestion for improvement. At the very least, all of the -> None hooks could be simplified in this fashion (and probably the others, too). I'm happy to put up a patch for that, if there's a process for dealing with a breaking change in the plugin API.

A specific reason I disliked the original pattern is that the provided fullname often does not have enough information for the hook to decide whether it cares, so the hook has to return a callback always. And yet, we spend cycles specifically extracting & passing just the fullname. For this reason, my base class hook is just return base_class_callback.

[snarkmaster]

As far as real-world usage of def mro, I have not yet found a big public project that heavily relies on it. But, it's not an unpopular idea:

• Zope uses an mro()-enabled metaclass to provide a backwards-compatibility shim: https://github.com/zopefoundation/ExtensionClass/blob/825fbf48a01eaf70d273ca9e9175f7465b38ba14/src/ExtensionClass/__init__.py
• An mro()-enabled metaclass to paper over a design bug in Django: http://www.memonic.com/user/danilo/folder/django/id/1xmoo
• Another person who needed it: https://stackoverflow.com/questions/20822850/change-python-mro-at-runtime
• A few people found this feature amusing enough to write about: http://stupidpythonideas.blogspot.com/2015/12/can-you-customize-method-resolution.html http://pybites.blogspot.com/2009/01/mro-magic.html
• CPython's test suite covers it well: https://github.com/python/cpython/blob/master/Lib/test/test_descr.py

[snarkmaster]

Also, if you have an idea for how I can type-check my metaclass instances without touching core mypy, I'd love to hear it. The "Example" story in the issue description is complete and accurate.

My primary goal is to get my project typed :)

[gvanrossum]

My primary goal is to get my project typed :)

And yet you don't appear to link to your project, so your concrete needs are pretty vague -- you gave an elaborate example, but it's not code, and I don't follow every part of it. In particular "At runtime, M.mro() can simply remove the A-produced base from B's MRO, and all is well" sounds mysterious -- who is calling mro() here?

You didn't help your case by combining it with a separate API change proposal.

Given that it's now August, maybe you no longer care, and then I would respectfully propose to just close this issue and the PR.

[snarkmaster]

Point of order: I think your comment would have read better without "And yet" and without "You didn't help your case". Factual and "I" statements cause less friction.

Project & specific needs

Due to the silence on this task, I put the typing work on hold.

What is the project-specific reason?

The MyPy-related code in question is not committed anywhere public, because I put it on hold. The code that is public has nothing to do with my request. I built a working prototype in a private branch, but for the purposes of the discussion you probably don't want to be reviewing 1000s of lines of code when 10s will suffice.

In words, the piece of code that needs the MyPy hook is a loose analog of a NamedTuple or a Pyrsistent PRecord or a frozen dataclass. My thing supports C3 inheritance for composing fields. Let's not debate too heatedly whether that's a good idea, given that it seems to serve well in at least my current codebase :)

My reason for MRO customization is as follows. The core implementation of these "frozen classes" is supplied by an implementation base class, which the metaclass injects on each of the "frozen classes" in the hierarchy. But, only the last-in-the-MRO implementation is needed, so I overload mro() to discard the earlier ones. Thus, I also need to hide those implementation bases from MyPy, since they would otherwise break type inference and creating spurious errors.

If you're willing to discuss adding an MRO customization hook to MyPy, then I will extract a minimal code example of how I am doing MRO customization. Let me know.

Request for MyPy

Regardless of the specific reason, at the end of the day, MyPy currently assumes that the MRO of a class is as-declared. And that's not the case with my metaclass. A hook of this sort in MyPy would be necessary and sufficient.

Separate API change proposal

If that drive-by suggestion is annoying to you, I'm happy to stick with the current API. I saw an opportunity for improvement and pointed it out. Do with it what you will.

This is still relevant

As you well know, type-checks are never "urgent" or "mandatory". That said, I would love to have working type-checks.

If you confirm the concrete list of changes you want to see on the PR, I will gladly update it.

[gvanrossum]

Fine. Submit the minimal PR that follows the established style and allows you to solve your problem. Someone will review it. Understand that the plugin API is not frozen or documented and can (and will!) change without notice.

[snarkmaster]

Thank you!

• I updated PR Add a customize_class_mro plugin hook #4567. Specifically:
  • I switched from my simple API of customize_class_mro to the more complex local convention of get_customize_class_mro_hook. If you see merit in my simplification, I can open a separate PR for it.
  • I rebased onto master of python/mypy and resolved conflicts.
• I look forward to changes in this API. I expect to need to fix my code. Luckily, this stuff is easy to unit-test.