[Support] catalystproject-latam, plnc: Add GPU profile (1 GPU + 4 GPU option)

[consideRatio]

The Freshdesk ticket link

https://2i2c.freshdesk.com/a/tickets/1591

Ticket request type

Feature Request

Ticket impact

🟩 Low

Short ticket description

We should enable the community to start servers with 1 and 4 GPUs respectively:

• n1-highmem-4 + 1 nvidia-tesla-T4 GPU (for small workloads)
• n2-highmem-16 + 4 nvidia-tesla-T4 GPUs (for large workloads)

Related documentation

• https://infrastructure.2i2c.org/howto/features/gpu/

Additional notes

Practically, I propose a dedicated profile_list entry for GPU, where the machine type + GPU count is determined by a option under the profile.

The image choices etc for this profile isn't specified by the community or me yet, so I propose a call is made and we ask for feedback.

Below are examples of how I think we need to declare kubespawner_override for the options choosing the hardware within the GPU profile

          kubespawner_override:
            # also here should be cpu/memory requests/limit to fill up the node picked
            environment:
              NVIDIA_DRIVER_CAPABILITIES: compute,utility
            node_selector:
              node.kubernetes.io/instance-type: n1-highmem-4
            extra_resource_limits:
              nvidia.com/gpu: "1"

          # .....

          kubespawner_override:
            # also here should be cpu/memory requests/limit to fill up the node picked
            environment:
              NVIDIA_DRIVER_CAPABILITIES: compute,utility
            node_selector:
              node.kubernetes.io/instance-type: n1-highmem-16
            extra_resource_limits:
              nvidia.com/gpu: "4"

(Optional) Investigation results

No response

[jmunroe]

Thanks @consideRatio . Your proposal for giving access to appropriately sized CPUs/GPUs looks good to me. Please make the changes.

[AIDEA775]

@consideRatio I have added the GPU options to two new image options. Did I understand your proposal correctly?

Also, the Pulling image "quay.io/pangeo/pytorch-notebook:2024.05.07" phase takes a lot of time.

[fvillena]

Can you add the new machines to the "Bring your own image" option please?.

And we don't need Pangeo, because we don't do geoscience. Can you replace those images with Jupyter-pytorch and Jupyter-tensorflow?

I think these are the correct images:

• quay.io/jupyter/pytorch-notebook
• quay.io/jupyter/tensorflow-notebook

What is the difference between Jupyter and Pangeo? because I spawned a Pangeo notebook, but it looks identical, but with Dask added? (and we'd like to start exploring Dask)

And please make sure cuda is working correctly

Thanks.

[fvillena]

And also, the spawn sometimes fails because:

Spawn failed: pod plnc/jupyter-fvillena did not start in 600 seconds!

[yuvipanda]

@AIDEA775 is there something I can do to help move this forward?

[consideRatio]

What is the difference between Jupyter and Pangeo? because I spawned a Pangeo notebook, but it looks identical, but with Dask added? (and we'd like to start exploring Dask)

The two projects independently maintain separate sets of images with several differences (at pangeo-data/pangeo-docker-images and jupyter/docker-stacks).

• Both provide images with pytorch and tensorflow installed.
• I think the pangeo images include misc cloud related CLI's to work with object storage
• I think the pangeo images include dask related software in all non-mimial images
• Overall, I think pangeo images include a bit more, while the jupyter images are a bit more tightly scoped

---

@fvillena we are working to get the jupyter images work with cuda fully in the cloud environment in #4269 - currently pending an upstream contribution. The pangeo images already work with cuda fully, so if you want dask + gpu, the pangeo image variants for tensorflow + pytorch could be reasonable to use instead and setup quickly.

[fvillena]

@fvillena we are working to get the jupyter images work with cuda fully in the cloud environment in #4269 - currently pending an upstream contribution. The pangeo images already work with cuda fully, so if you want dask + gpu, the pangeo image variants for tensorflow + pytorch could be reasonable to use instead and setup quickly.

Thanks for the reply. I will be using the Pangeo images until the Jupyter images work with your infrastructure.

But I still need you to add the new machines to the "Bring your own image" option.

[consideRatio]

[...] add the new machines to the "Bring your own image" option.

@AIDEA775 has worked to refine the options already, you can now select your own image no matter if you opt for CPU only or With GPUs.

Does this look alright?

[consideRatio]

I will be using the Pangeo images until the Jupyter images work with your infrastructure.

With jupyter/docker-stacks#2124 merged, the jupyter images now work with GPUs as well.