[k8s] Enable multiple kubernetes contexts for failover

[Michaelvll]

This allows user to specify the following ~/.sky/config.yaml to enable SkyPilot to failover through different kubernetes contexts.

kubernetes:
  allowed_contexts:
    - kind-skypilot
    - gke_skypilot-xxx_us-central1-c_test-zhwu

TODO:

• Check if we should improve the UX output for region vs context
• Multiple k8s clusters with different resources
• show-gpus should show resources from all allowed_contexts (left for future)
• Add example policy for dynamic k8s context update
• Add tests for multi-k8s

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • sky launch -c test --cloud kubernetes --cpus 4 echo hi with two k8s clusters, one with nodes having less than 4 CPUs, one with nodes with more than 4 CPUs; it correctly failover through the first k8s cluster to the second one
  • Remove the larger k8s cluster context name from allowed_contexts, and sky exec/ sky launch again on the existing SkyPilot cluster
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[romilbhardwaj]

Thanks @Michaelvll! Took a quick look

[romilbhardwaj]

[commentary, no action required] I am liking the idea of using regions (instead of clouds) to do multi-kubernetes. In the future, if we want to enable multi-k8s out of the box, we can simply return all contexts here :)

[Michaelvll]

Conceptually, I found it more clear to have the following mapping:
k8s contexts -> local cloud config profiles.
Because both of them contains:

1. the identity to use for accessing the resource pool (k8s: user + namespace; cloud config: account)
2. the resource pool to look at (k8s: cluster; cloud config: project to use)

I think the current way is a simple workaround for now, but we may need to have a better design in the future. The main confusion with using region may come from: multiple context can map to the same k8s cluster with different namespace or user.

[romilbhardwaj]

Agreed, probably need better solutions. I just realized many properties in the config may need to be updated in the near future to work well for multi-cluster (e.g., some contexts may need ports: ingress, while others may need ports: loadbalancer. Same ofr other fields).

[Michaelvll]

Yes, when I am updating the code for always showing context for region, I also realized that there are more places to be updated, especially the code for failover Kubernetes._get_feasible_launchable_resources. If we have two clusters with different resource set, our failover will likely disregard all the Kubernetes clusters if the cluster without the resource is the current activate context.

Marking this PR to draft for now to fix this issue.

[romilbhardwaj]

Thanks @Michaelvll! Tested it and works nicely. Left some comments.

[romilbhardwaj]

Agreed, probably need better solutions. I just realized many properties in the config may need to be updated in the near future to work well for multi-cluster (e.g., some contexts may need ports: ingress, while others may need ports: loadbalancer. Same ofr other fields).

[Michaelvll]

We realized that we need to update the code for checking resource feasibility on a kubernetes cluster to support different context and make failover fully functional. Changed this PR to draft for now to fix that issue.

[Michaelvll]

Fixed the feasible resources checking and seems working with multiple kubernetes containing different resources.

Test setup

sky local up and another GKE cluster with 2 nodes.

1. Local k8s: labeled with L4 GPUs, with test-namespace as namespace
2. GKE cluster: 1 node labeled with H100 GPU, with gke-namespace as namespace

• local k8s as current context, sky launch --gpus h100 echo hi -- correctly go to GKE and launch a machine
• GKE as current context, sky launch --gpus h100 echo hi -- correctly go to GKE and launch a machine
• no allowed context specified, GKE as current context: sky launch --gpus h100 echo hi -- correctly go to GKE and launch a machine
• no allowed context specified, local k8s as current context: sky launch --gpus h100 echo hi -- does not launch on GKE
• Two allowed contexts, create a cluster on local k8s cluster, remove the kind-skypilot from allowed_contexts and exec again on the previous cluster.

TODO:

• backward compatibility
  • On master, sky launch -c test --cloud kubernetes echo hi switch to this PR sky exec test echo hi
  • sky status -r test
  • sky launch -c test echo hi again
• Add a smoke test

[romilbhardwaj]

This is awesome, thanks @Michaelvll! Tested failover on GKE + local. Left some comments

[Michaelvll]

This should be ready for another look. @romilbhardwaj : )

Future TODOs:

• Add a dedicated doc for multiple kubernetes

[romilbhardwaj]

Since this test will fail if the sky local up cluster is not setup, can we add a quick error message for the dev running this test at this line? Something along the lines of: "Unable to find mocked GPUs in the sky local up cluster. Please read the instructions for test_kubernetes_context_failover on how to set it up".

Or better yet, automate the sky local up setup :)

[Michaelvll]

Added. Thanks!

[Michaelvll]

Automatic local up is a bit scary, as when I tried to do it, it turned out to me that we may have multiple tests in the future using the same local k8s cluster, and can cause issue if everyone is trying to modify that cluster.

[romilbhardwaj]

Awesome work @Michaelvll! LGTM.