REF: Make Util functions accessible from C

[WillAyd]

Came up when reviewing #50400 with @lithomas1 . Doesn't directly solve this, but a lot of our util.pyx functions are in essence C functions to begin with. Because they are implemented in Cython though, its a built difficult to use them in our non-Cython extensions like JSON.

This takes the code out of the Cython file and ports to a C header file. This is mostly a verbatim copy with some special considerations done for NA handling. Marking as draft because I'm not sure a header-only solution is better than a shared library, but need more time to mess around with the setup script. May also just wait until we implement meson fully

[WillAyd]

Initial timeits don't show anything conclusive between main:

In [5]: %timeit lib.is_integer_array(np.array([1, 2]))
525 ns ± 0.77 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [6]: %timeit lib.is_integer_array(np.array([1, 2]))
466 ns ± 1.64 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [7]: %timeit lib.is_integer_array(np.array([1, 2]))
469 ns ± 1.51 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [8]: %timeit lib.is_integer_array(np.array([1, 2]))
468 ns ± 0.787 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

and this PR

In [7]: %timeit lib.is_integer_array(np.array([1, 2]))
531 ns ± 0.679 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [8]: %timeit lib.is_integer_array(np.array([1, 2]))
479 ns ± 1.41 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [9]: %timeit lib.is_integer_array(np.array([1, 2]))
474 ns ± 3.77 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

[WillAyd]

Not sure the implementations ulltimately should be in this header file. Done for now just to emulate the existing util.pxd file, but would make the build system a bit easier with a dedicated .c implementation

[WillAyd]

type_inference.h might also be a better name than just type.h

[jbrockmendel]

IIRC cython has some mechanism to make cython functions available in C. is that an option?

[WillAyd]

Looks like this is the section in the docs:

https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#using-cython-declarations-from-c

Worth trying in the future for more advanced use cases, but I think here the functions we are putting into Cython functions are really C functions anyway, so would be less steps the way it is

That said, I think this PR will be dependent on moving away from setuptools into meson. setuptools it seems implicitly requires that all extensions you build are actually Python extensions, so we aren't able to really set up the full dependency chain needed without meson

[lithomas1]

IIRC cython has some mechanism to make cython functions available in C. is that an option?

IMO, ideally, we should be doing C -> Cython since its much easier from a build standpoint unless absolutely necessary (Cython generates the C header files at runtime when going the other way which can be tricky).

[lithomas1]

Looks like this is the section in the docs:

https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#using-cython-declarations-from-c

Worth trying in the future for more advanced use cases, but I think here the functions we are putting into Cython functions are really C functions anyway, so would be less steps the way it is

That said, I think this PR will be dependent on moving away from setuptools into meson. setuptools it seems implicitly requires that all extensions you build are actually Python extensions, so we aren't able to really set up the full dependency chain needed without meson

This should be possible with build_clib. (See https://github.com/pandas-dev/pandas/pull/47081/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7).

The only question left should be whether you want to wait for setuptools support to be dropped. Assuming the meson build system PR lands sometime in January, we'll probably have to wait until no more 2.0.x releases to fully drop setuptools to prevent main/2.x from diverging too much. (I guess we can push the transition more aggressively/delay 2.0 slightly after the PR lands too. At any rate, we'd probably need 1 month with the new meson build system before I'm confident enough to drop setuptools)

[WillAyd]

Yea I think this should wait until we drop setuptools. There's no rush on this PR and I'm not confident we can do it well without meson

[jbrockmendel]

Not a huge deal, but we specifically moved the util.pxd functions from a now-removed numpy_helper.h file years ago to be more friendly to non-C contributors

[WillAyd]

Gonna close for now - dependent on the new build system. Can come back after that work stabilizes