PSA: Externally managed Istio 1.25.3 tested as working for deployKF 0.15.0 (+tips pre EKS 1.31 upgrade)

[berglh]

EKS 1.30 is entering extended maintenance next week, and prior to updating to 1.31, we upgraded Istio to version 1.25.3 (that officially supports Kubernetes 1.31). #254 also shows that EKS 1.31 is working with deployKF.

ℹ️We are only running the Kubeflow notebooks app currently and not using Pipelines and ML compoments. Please ensure you check the istioctl analysis commands before proceeding.

We also run our own Istio Ingress Gateway deployment, let deployKF manage its Istio Gateway resource and have a separate Istio Gateway resource for our other apps like self-managed ArgoCD and Airflow. We also run the Istio CNI chart for the cluster but have left the sidecar operation for deployKF components.

Pre-Upgrade

ℹ️Before upgrading Istio, it's always recommended to use istioctl to check for upgrade compatibility issues. Download the latest version of the istioctl: Install istioctl. Once installed, run the following commands:

istioctl analyze
istioctl x precheck

Check the output for any issues, take careful note of any compatibility issues, particularly if they may impact the operation of deployKF & Kubeflow.

Issues

When I updated the Helm chart for Istio we deploy with Terraform, we ran into a few minor issues:

1. The ISTIO_META_DNS_AUTO_ALLOCATE proxyMetadata setting in meshConfig is deprecated, we removed this from the primary Istio chart.
2. Remove platform=aws from our Istio gateway chart (it was an invalid value, standard chart works).
3. Adjusted rolling strategy for the Istio gateway chart:

   strategy:
     rollingUpdate:
       maxSurge: 0
       maxUnavailable: 1
   

Other Notes

• After upgrading Istio Base, Istiod, Istio CNI, and Istio Ingress Gateway deployments, the access to dex/Oauth2 broke. Simple restarting all the pods for the deploykf-auth namespace resolved the issue.
• It's probably worth adjusting the Istio update strategy for other Istio charts, I had to manually restart/delete pods for the Istiod deployment.