CI: Investigate using build caches such as `sccache-action`

[leofang]

I learned that https://github.com/Mozilla-Actions/sccache-action is a thing. If possible we should use a build cache like this to accelerate the build stage. Building cuda-bindings is a bit slow and we don't make changes to cuda-bindings in every PR, yet they are rebuilt on a per-PR basis.

[rwgk]

I'm all for this, but I've also been wondering: Could we use a different pip build tool to get more efficient parallel builds? (@cpcloud?)

When I played with pytorch pip install a couple months ago, I saw they achieve very efficient parallel builds. It automatically uses all cores. Could we learn from that?

[cpcloud]

I think we have to determine what the source of the slowness is before making a particular choice.

Here's a back-of-the-napkin analysis of our current state.

Generally speaking there are only a few kinds of large actions we can take to speed things up, though I'm sure @cryos has lots of thoughts here:

1. Making files smaller so that more TLs can be compiled at once
2. Avoiding recompilation (usually via some kind of caching). This can be at the TL level or at the package (aka dependency) level.

Our current compilation path looks like this (NOT including any cybind/cython-gen steps, i.e., assuming that all pyx/pxd are ready to be compiled to C):

1. Compile Cython to C (sped up by making cython faster, or compiling smaller files and parallelizing)
2. Compile C to object files (very slow, likely because the generated files are enormous, also sped up by splitting TLs).
3. Link object files (also very slow, but this is a somewhat notoriously difficult problem to speed up). The mold linker could be something worth exploring here.

Something like sccache would very likely help with step 2 (and maybe step 3, though not sure about that since I don't think you can swap in newer bits into an already-linked object), since in theory only things that changed (transitively) would be recompiled.

At some point linking will become the bottleneck, e.g., when all but one object file is hit in the cache and the entire thing needs to be relinked.

All that is to say, I don't think anything in the Python build tool space is going to help us here since AFAIK none of those tools directly improve any of the fundamental bottlenecks we see.

[cryos]

Yes this all matches my thinking, I know less about step 1. I would add that moving to scikit-build-core and cmake would open up the door to a lot of standard tooling. Ninja is in general going to make best use of all cores, sccache is already available to our build runners, it is easy to install with the action mentioned or baking it into images. If linking is a bottle neck I completely agree mold is a good (and pretty easy) way to go but it can have side effects.

Many projects use cmake, ninja and sccache within RAPIDS and other NVIDIA org projects. There is already an issue to move to scikit-build-core, it is something I would like to do. I don't know the custom Python build system used that well but it would likely be able to use sccache in the short-term and is a quick thing to add in all likelihood.

[cpcloud]

I don't think any of the build systems here are special in that they when they shell out to the compiler, it's done in the usual way, by using a slightly modified version of build_ext/bdist_wheel.

The only heavily customized build is for cuda_bindings and I think that kind of thing can be done independent of use of scikit-build-core and cmake and whatever other tool we may want to use.

That customized step is important (generating cython files), but I think it can be preserved as needed.

From the build engineering perspective I see it as this:

1. Something produces cython files
2. scikit-build-core takes over

At this point I suspect it's more a matter of "who has cycles to work on this" than anything else.

[leofang]

Phillip's analysis above is largely correct. Though there's a Step 0 that's missed (but later noted), which is populating the Cython pxd/pyx files from the template files.

Step 0 needs to be fixed (it's very slow too) but it requires changes to the codegen (https://github.com/NVIDIA/cuda-python-private/issues/149), which is orthogonal to the discussion here. Then, Step 2 is really the slowest step from my experience. We generate cpp files that can be as long as 200k lines or more IIRC. It's a lot of code that we're compiling, and sccache will definitely help.

I do not believe we need to update the build system to improve the build time, I agree with Phillip that we want to do it for other reasons but not this (#605), because ultimately it's gcc/msvc being slow.