Complete abi3 support

[alex]

abi3, also known as pylimited abi, is a feature of built python projects where they can be loaded from multiple versions of python3. pyo3 has some limited support for this, but it's commented out in Cargo.toml because it's currently incomplete and does not compile.

I'm filing this as a tracking bug so folks interested in abi3 support have a place to follow along. I intend to attempt to contribute to resolving this issue.

I imagine this is a three part process:

1. switching things from internals -> abi3 APIs when there's a 1:1 equivalent
2. disabling pyo3 APIs that map to things that don't exist in abi3 mode
3. figuring out how to re-enable the things from (2)

[kngwyu]

You need #[pyclass] with abi3?
I think most of the code works with PY_LIMITED_API or can be modified to do so, but not sure about #[pyclass].

[davidhewitt]

I think it would be desirable if #[pyclass] can work with the limited API, but we may need to talk a lot to upstream cpython to achieve it.

[alex]

Why do you imagine it'd require upstream support? I imagine most #[pyclass] usage can be achieved with PyType_FromSpec?

[davidhewitt]

Quite possibly - might need a lot of refactoring!

[alex]

I started investigating #[pyclass] support, since it's the biggest piece here. Before I get too deep into implementing, I wanted to lay out my thoughts and make sure this all sounded ok.

First, I'll start by describing the current approach:

• Currently pyo3-dervice-backend emits code that creates a register, which contains a struct for each of the tables of function pointers that a PyTypeObject* contains.
• pyo3 then takes this table, create a PyTypeObject* and fills in all the fields by hand basically, and then calls PyType_Ready.

My proposal for how to make this abi3 compatible:

• pyo3-dervice-backend is changed to generate a list of PEP384 slots
• pyo3 takes those and calls PyType_FromSpec

The one challenge is that some of these APIs are technically public (though they often have #[doc(hidden)]) because they're called from macros that expand in users's crates. Is it ok to change these APIs - perhaps with a major version bump?

One of my goals here is that there's only one version of this code, and not 2 different versions.

[alex]

#1132 has a PoC of what the first steps of this could look like