AKS LoadBalancer IP address changed when upgrading from v2 to v3

[jannekem]

Describe the bug
After upgrading NGINX Ingress Controller my services became unavailable as the LoadBalancer IP address changed on an Azure AKS cluster.

I couldn't find any mention about such a possible side effect from the release notes.

To Reproduce
I have reproduced this on two AKS clusters. My services are exposed with VirtualServers. After I run helm upgrade the VirtualServer status.externalEndpoints still shows the old IP address but the service is not reachable anymore. The new IP address can be found from Azure Portal in the "Services and ingresses" page of the AKS cluster instead.

Here are the versions I'm upgrading from/to.

$ helm history ingress-controller
REVISION        UPDATED                         STATUS          CHART                   APP VERSION     DESCRIPTION     
5               Fri Nov 18 09:10:41 2022        superseded      nginx-ingress-0.15.1    2.4.1           Upgrade complete
6               Wed Apr 12 09:38:06 2023        deployed        nginx-ingress-0.17.0    3.1.0           Upgrade complete

Expected behavior
The IP addresses should not change during an upgrade.

Your environment

• Version of the Ingress Controller 3.1.0
• Version of Kubernetes 1.25.5
• Kubernetes platform AKS
• Using NGINX

Additional context
Helm settings:

controller:
  enableSnippets: true
  enableCustomResources: true
  enableCertManager: true
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 1000m
      memory: 1Gi

[github-actions]

Hi @jannekem thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

[coolbry95]

I wonder if this is because the name of the service changes.

[haywoodsh]

Hi @jannekem, thank you for reaching out to us. I am not sure that there is a guarantee that the IP address won't change during a helm upgrade. If you have upgraded to version 2.4.1 from an earlier version before, could you please let me know if you have also experienced such a problem?

[jannekem]

With the production cluster we've gone through the following versions: 1.8.1 -> 2.0.0 -> 2.0.3 -> 2.3.0 -> 2.3.1. The IP address has stayed the same through all these revisions.

I upgraded the development cluster from 2.3.1 and the test cluster from 2.4.1, and they both had the same behavior. I haven't upgraded the production cluster yet because that would require coordination to update the DNS records if/when the IP changes.

[De1taE1even]

OP is correct about this issue. I'm looking the chart helper, and the naming logic has changed. This is the parameter I have been using (and I assume OP is using):

controller:
  service:
    name: "nginx-ingress"

The original service was created with the name nginx-ingress. Cloud services (Azure in my case) provision an IP address when the service is created. With the new naming logic, the helm chart wants to delete the current service and create a new one named nginx-ingress-controller. With your new helper logic, I do not see a way to remove that trailing -controller naming, because you're assembling all names off of the .Values.controller.name parameter now.

This is very bad for customers like me because I cannot afford the caching issues and potential downtime involved with an IP address change to our primary operational ingress.

[haywoodsh]

Thank you for your input, @jannekem and @De1taE1even. While the change in naming logic for the service was made to align with Helm best practices, I understand that it may have caused unexpected behaviour for users. We are currently investigating any other unforeseen consequences that this change may have introduced and will work on improving the upgrade experience for those affected by this issue.

[timnee]

I'm experiencing the same issue in AWS. The service name change causes the ELB to be replaced. In my case, I'm able to keep the service name from changing by setting .Values.controller.name. However, that also keeps the deployment name from changing. This causes the upgrade to fail since spec.selector.matchLabels is immutable. The name of both the service and deployment are from nginx-ingress.controller.fullname. There is currently no way to keep the service name and upgrade the deployment. Adding a service name override would solve my issue.

[coolbry95]

We have a similar issue where all resource names change. Before most resources were a simple "nginx-ingress" now they are "something-nginx-ingress". This is unavoidable even with fullNameOverride and nameOverride. You will always end up with "something-nginx-ingress" for some resources. For example if we set fullNameOverride to "nginx-ingress" some resources will be named "nginx-ingress" and others will be named "nginx-ingress-nginx-ingress".

Maybe its out of scope for this issue but I think its falls inline. Should overrides be provided for all resources?

[lucacome]

Just wanted to let everybody know that we merged the fix created by @timnee and will add it to our upcoming release (v3.1.1) with a note in the changelog on how to use it. Let me know if you have any thoughts/suggestions about it.

@coolbry95 you can always remove the name with --set controller.name="", does that help?