How do I get a single Ingress to be handled by just one of two HAProxy Ingress Controllers on AKS?

[cdhunt]

Full Question: https://stackoverflow.com/questions/68580249/how-do-i-get-two-haproxy-ingress-controllers-to-handling-a-single-ingress-on-aks

tl;dr. I have two ingress controllers. I'm using ingressClassName to associate Ingress resources with one or the other. However, I only have one Ingress set up configured with one ingressClassName but both controllers are servicing requests.

[bluebrown]

have you tried giving the second one a proper class instead of --empty-ingress-class?.

As far as I understand, with this flag it will handle all ingress resources with and without class-name. I think its meant to use when you have only 1 controller that should handle everything and don't want to add that class everywhere.

But the way you try it would also make sense and be good IMO. Maybe there should be a third flag, handle-all-ingress or something.

[cdhunt]

Yeah, the internal Ingress has a class and the empty class argument. The external just has the class argument and it still seems to handle all requests regardless of class.

[Mo3m3n]

Hi @cdhunt
When I run Ingress Controller with --ingress.class set to "internal-lbit only deals with Ingress resources havingkubernetes.io/ingress.class: internal-lb` so it works as expected and I can't reproduce your issue.
Can you provide the following:

• Ingress Controller version
• Ingress Controller params (command line args) for both controllers

[cdhunt]

Thanks for the follow-up. I'm out this week. I'll post those details next week.

[cdhunt]

External

           ...
          image: haproxytech/kubernetes-ingress:1.6.5
          args:
            - --configmap=ext-haproxy-controller/haproxy
            - --default-backend-service=ext-haproxy-controller/ingress-default-backend
            - --disable-ipv6
            - --ingress.class=external-lb
            ...
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: external-lb
spec:
  controller: haproxy.org/ingress-controller
  parameters:
    apiGroup: k8s.example.com
    kind: IngressParameters
    name: external-lb

Internal

          ...
          image: haproxytech/kubernetes-ingress:1.6.5
          args:
            - --configmap=haproxy-controller/haproxy
            - --default-backend-service=haproxy-controller/ingress-default-backend
            - --disable-ipv6
            - --ingress.class=internal-lb
            - --empty-ingress-class
            ...
---
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: internal-lb
  annotations:
    "ingressclass.kubernetes.io/is-default-class": "true"
spec:
  controller: haproxy.org/ingress-controller
  parameters:
    apiGroup: k8s.example.com
    kind: IngressParameters
    name: internal-lb

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    "kubernetes.io/ingress.class": internal-lb
spec:
  ingressClassName: internal-lb
  rules:
  - host:  oursite.dev
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: site
            port:
              number: 80

[cdhunt]

Also confirmed that both controllers have the same backends configured and the controllers are reading the arguments.

2021/08/16 13:54:02 ConfigMap: haproxy-controller/haproxy
2021/08/16 13:54:02 Ingress class: internal-lb
2021/08/16 13:54:02 Empty Ingress class: true
---
2021/08/16 13:54:14 ConfigMap: ext-haproxy-controller/haproxy
2021/08/16 13:54:14 Ingress class: external-lb
2021/08/16 13:54:14 Empty Ingress class: false

[sniip-code]

@cdhunt I think the issue is this class func igClassIsSupported()

kubernetes-ingress/controller/ingress.go

Line 32 in d103eee

func (c *HAProxyController) igClassIsSupported(ingress *store.Ingress) bool {

If IngressClassName is set it matches on all controllers as they are all "haproxy.org/ingress-controller".

I fixed it by removing "ingressClassName" annotation in the ingress.yaml. The legacy code works only matching on controllers with matching ingress.class.

So just remove ingressClassName and leave "kubernetes.io/ingress.class" set with your className matching the className in the haproxy startup args.

I also found it useful to port forward to the stats web interface and see what backend services have been mapped. Before I removed IngressClassName I had backend services from other ingress files.

kubectl port-forward "nameOfIngressPod" 1024

The go to: http://127.0.0.1:1024

Before and after removing "IngressClassName" check the web interface only has backend services for the services in the ingress. When I had "ingressClassName" annotation set there were backend services from other ingress files, when i removed it only had the correct backend services.

Tell me how you go..

Cheers,

Internal

image: haproxytech/kubernetes-ingress:1.6.5
args:
  - --configmap=ext-haproxy-controller/haproxy
  - --default-backend-service=ext-haproxy-controller/ingress-default-backend
  - --disable-ipv6
  - --ingress.class=external-lb
            ...

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    "kubernetes.io/ingress.class": internal-lb
spec:
  rules:
  - host:  oursite.dev
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: site
            port:
              number: 80

External

image: haproxytech/kubernetes-ingress:1.6.5
args:
  - --configmap=ext-haproxy-controller/haproxy
  - --default-backend-service=ext-haproxy-controller/ingress-default-backend
  - --disable-ipv6
  - --ingress.class=external-lb
  ...

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    "kubernetes.io/ingress.class": external-lb
spec:
  rules:
  - host:  other.dev
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: site
            port:
              number: 80

[cdhunt]

@sniip-code That was it and I think the only combination I didn't try. Thank you!

To summarize, ingressClassName with IngressClass doesn't work when you have multiple haproxy controllers. I assume it works fine when the controllers are different controller types.

[cdhunt]

@sniip-code Do you want to add an answer to https://stackoverflow.com/questions/68580249/how-do-i-get-a-single-ingress-to-be-handled-by-just-one-of-two-haproxy-ingress-c?

[Mo3m3n]

Thanks @sniip-code for taking the time to look into this issue !
@cdhunt on my first post I have missed the ingressClassName test so I was not able to replicate this before reading @sniip-code message
To make things clearer there are two different methods to assign Ingress resources to Ingress Controllers.

• IngressClass Kind: relying on IngressClass resource and IngressClassName field)
• ingress.class annotation: relying on the controller argument --ingress.class

The first one is the officially supported method and the second one is the legacy one.
We support both of them, but here we have a buggy behavior with IngressClassName that needs to be fixed.

To summarize, ingressClassName with IngressClass doesn't work when you have multiple haproxy controllers. I assume it works fine when the controllers are different controller types.

To be more accurate the issue here is only with ingressClassName, our Ingress Controller is not currently processing the parameters field in the IngressClass but only the controller field. So if the controller field is matching, the ingress resource will be processed, which means all haproxy controllers will be involved without being able to differentiate them.
That's why when you remove ingressClassName the issue disappears and ingress.class is doing the job correctly .

---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: external-lb
spec:
  controller: haproxy.org/ingress-controller
  parameters:
    apiGroup: k8s.example.com
    kind: IngressParameters
    name: external-lb

I am currently working on providing a fix for this. Will keep you posted here as soon as it is pushed and released.

[sniip-code]

@sniip-code Do you want to add an answer to https://stackoverflow.com/questions/68580249/how-do-i-get-a-single-ingress-to-be-handled-by-just-one-of-two-haproxy-ingress-c?

I replied to your stackoverflowq question.

[gbarazer]

Hi,

I can confirm that bug when using two ingressclasses implementing 2 haproxy controllers. The function in

kubernetes-ingress/controller/ingress.go

Line 35 in 9034c1e

func (c *HAProxyController) igClassIsSupported(ingress *store.Ingress) bool {

only checks the class name in the annotation and only checks if the ingressclass controller matches haproxy, but does not check the name.

This makes it impossible to have an ingress serviced by only one of the controllers using only the ingressClassName spec, and would require to leave also the soon-to-be obsoleted ingress.class annotation which is the only value that the controller checks to decide if the ingress should be serviced by the controller. It seems to be impossible to have both ingressClassName and the ingress.class annotation (kubernetes.io/ingress.class) at the same time on an ingress on Kubernetes 1.22