404 on ACME challenge because of missing Ingress rules

[hahahannes]

Describe the bug
The created Ingress is not applied by the Controller. Therefore I get a 404 when I try to 'solve' the ACME challenge.

To Reproduce
Steps to reproduce the behavior:

1. Deploy NGINX ingress controller Version 2.4.1
   Helm installation with ingressClass: nginx

2. Deploy cert-manager Version 1.11

3. Setup an Issuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-staging
spec:
 acme:
   server: https://acme-staging-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     name: letsencrypt-staging
   solvers:
   - http01:
       ingress:
         class:  nginx

5. Create a Certificate resource (not an Ingress)

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cert
spec:
  secretName: cert
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging
  commonName: foo.bar
  dnsNames:
  - foo.bar

Expected behavior
An Ingress for ACME HTTP01 challenge should be created which should get picked up by the Controller.
The ACME URL should then be reachable.

Your environment

• Version of the Ingress Controller: 2.4.1
• Version of Kubernetes: 1.20
• Kubernetes platform (e.g. Mini-kube or GCP): self hosted
• Using NGINX or NGINX Plus: NGINX
• Cert-Manager: 1.11

Additional context

• Everything works fine when I use an Ingress resource with the cert-manager annotations.
• I double checked the Ingress Class of the Issuer and the created Ingress to match the class specified in the NGINX controller configuration.
• The Ingress has no Events to it when kubectl describe ingress is done.
• The Controller seems to notice that an ingress is created
• But It logs only: Using the DEPRECATED annotation 'kubernetes.io/ingress.class'. The 'ingressClassName' field will be ignored. A Normal/Error Event is missing.
  Detailed log with logLevel=3:

I0118 14:50:45.829372 1 handlers.go:197] Adding service: cm-acme-http-solver-n6w69
--
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.829418 1 task_queue.go:61] Adding an element with a key: default/cm-acme-http-solver-n6w69
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.829467 1 task_queue.go:94] Syncing default/cm-acme-http-solver-n6w69
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.829482 1 task_queue.go:73] The queue has 0 element(s)
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.829500 1 controller.go:882] Syncing default/cm-acme-http-solver-n6w69
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.829513 1 controller.go:2004] Syncing service default/cm-acme-http-solver-n6w69
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.838546 1 handlers.go:102] Adding Ingress: cm-acme-http-solver-4w2fv
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.838585 1 task_queue.go:61] Adding an element with a key: default/cm-acme-http-solver-4w2fv
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.838616 1 task_queue.go:94] Syncing default/cm-acme-http-solver-4w2fv
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.838630 1 task_queue.go:73] The queue has 0 element(s)
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.838654 1 controller.go:882] Syncing default/cm-acme-http-solver-4w2fv
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.838685 1 controller.go:1971] Adding or Updating Ingress: default/cm-acme-http-solver-4w2fv
Wed, Jan 18 2023 3:50:45 pm | W0118 14:50:45.838707 1 controller.go:3641] Using the DEPRECATED annotation 'kubernetes.io/ingress.class'. The 'ingressClassName' field will be ignored.
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.839115 1 controller.go:1227] Processing 0 changes
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.839135 1 controller.go:1187] Processing 0 problems
Wed, Jan 18 2023 3:50:45 pm | I0118 14:50:45.849809 1 handlers.go:68] Adding endpoints: cm-acme-http-solver-n6w69

• I already tried different ACMEChallengeSolverHTTP01Ingress as described in http-01 challenge propagation: wrong status code '404', expected '200' cert-manager/cert-manager#2517 (comment)
• This happened after I upgraded the NGINX Ingress controller (1.8.0 -> 2.4.1). An older cert-manager version (1.3) was used, too at that moment. So I upgraded it to 1.11 in the hope that it would work then
• I think this is not related to Allow ingressClassName to be set for HTTP01 solver ingresses. cert-manager/cert-manager#4821 as the current official NGINX controller supports both the annotation and the ingressClassName

[github-actions]

Hi @hahahannes thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this 🙂

Cheers!

[hahahannes]

Here the Ingress spec, that gets generated by cert-manager

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  creationTimestamp: "2023-01-19T10:31:13Z"
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    acme.cert-manager.io/http-domain: "2558134934"
    acme.cert-manager.io/http-token: "691867083"
    acme.cert-manager.io/http01-solver: "true"
  managedFields:
  - apiVersion: networking.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubernetes.io/ingress.class: {}
          f:nginx.ingress.kubernetes.io/whitelist-source-range: {}
        f:generateName: {}
        f:labels:
          .: {}
          f:acme.cert-manager.io/http-domain: {}
          f:acme.cert-manager.io/http-token: {}
          f:acme.cert-manager.io/http01-solver: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"ab5644d5-bbef-43dd-9a49-f2c9346fa7d7"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:rules: {}
    manager: cert-manager-challenges
    operation: Update
    time: "2023-01-19T10:31:13Z"
  name: cm-acme-http-solver-ffmx9
  namespace: default
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: cert-c9td5-2295133753-1950763553
    uid: ab5644d5-bbef-43dd-9a49-f2c9346fa7d7
  resourceVersion: "396309218"
  uid: 99c3b6e2-973e-4ba6-b050-85084cc71109
spec:
  rules:
  - host: foo.bar
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-bjppg
            port:
              number: 8089
        path: /.well-known/acme-challenge/-3CVLSKmxFUKT7EX5hwsAfVJ_-IjFQDg_HqEc7hRKdE
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

Also does not work when I add the new ingressClassName field manually

[shaun-nx]

Looking into this now.

[hahahannes]

Thank you very much for your help!
I am also wondering what role the cert-manager plays in this problem. I have other "normal" Ingresses that have the same configuration and they are added by the Ingress Controller

[shaun-nx]

Hi @hahahannes
Can you tell me what the value of the certManagerEnabled option is in your helm values?

[hahahannes]

Its enabled because I use the cert-manager field in a VirtualServer

[hahahannes]

Thanks a lot for the quick fix @ciarams87!
Is there a timeline when this will be released?