formalize the ROCm image pre-build process

[jeffdaily]

ROCm must build magma and miopen from source. This takes considerable time and also exceeds allowed disk space on the GitHub Action runner.

ROCm images are now a two-stage process. For example, first run:

GPU_ARCH_VERSION=5.5 ./manywheel/build_docker_rocm_prebuild.sh

This builds and pushes the image:

rocm/dev-centos-7:5.5-magma-miopen-staging

Then the GHA workflow can proceed as usual.

[malfet]

Please add CI to do that. Also, there are no harm in picking a bigger runner for the job (as Docker images are rebuilt very infrequently)
Also, just curios, why MAGMA built mechanism for ROCm and CUDA are diverged?

[jithunnair-amd]

@malfet Do you mean "add CI to build rocm/dev-centos-7:5.5-magma-miopen-staging" image? That docker image wouldn't be in a PyTorch docker repository.

@jithunnair-amd and that's a problem, isn't it? Because if somebody needs to fix/update magma, they'll have to ask you guys to update the basic container.

As for Magma, the CUDA flow installs magma from a conda package, while the ROCm flow builds magma from source.

Yes, I know the mechanics, but the question is: why? I.e. magma for CUDA is build by the following workflow, why can't rocm build of magma be added to it?

[jithunnair-amd]

@malfet Do you mean "add CI to build rocm/dev-centos-7:5.5-magma-miopen-staging" image? That docker image wouldn't be in a PyTorch docker repository.

@jithunnair-amd and that's a problem, isn't it? Because if somebody needs to fix/update magma, they'll have to ask you guys to update the basic container.

I will say that we actually considered using one of our ROCm GHA runners for the builder CI jobs, but picking a bigger CPU-only GHA runner seems like a fine idea too. @jeffdaily, should we try the bigger GHA runner to see if we can avoid this two-stage docker build process?

As for Magma, the CUDA flow installs magma from a conda package, while the ROCm flow builds magma from source.

Yes, I know the mechanics, but the question is: why? I.e. magma for CUDA is build by the following workflow, why can't rocm build of magma be added to it?

Actually, we have been working on this recently, trying to take a look at the CUDA magma build flow to see if we can apply it to ROCm magma as well. It's a WIP.

[malfet]

@jithunnair-amd I'm not saying that ROCm magma builds need to follow CUDA magma builds or vice versa, but it would be nice if workflows can be kept as close to each other as possible.

[jeffdaily]

I will say that we actually considered using one of our ROCm GHA runners for the builder CI jobs, but picking a bigger CPU-only GHA runner seems like a fine idea too. @jeffdaily, should we try the bigger GHA runner to see if we can avoid this two-stage docker build process?

YES! That would fix our issues, too, without all this pre-build trickery.

[jithunnair-amd]

@malfet @seemethere What are the runner instance names to use for larger diskspace/CPU? This doesn't indicate specific labels: https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners#machine-specs-for-larger-runners?

[malfet]

@jithunnair-amd we have self-hosted runners running on AWS which are defined in https://github.com/pytorch/test-infra/blob/main/.github/scale-config.yml

[jeffdaily]

Closing in favor of #1418.