Unable to remove or update the NGINX added X-Forwarded-Proto header using `charts/ingress-nginx`

[ashageetha]

What happened:

Cannot remove or override the nginx ingress controller added X-Forwarded-Proto header value. Even though able to pass the customized proxy header using below values.yaml to helm install, but the URL still gets added with X-Forwarded-Proto: http header value.

controller:
  proxySetHeaders:
    X-Forwarded-Proto: ""

The nginx.conf inside the ingress controller pod shows the customized value is being added but the URLs still gets added with the first value and not the latest X-Forwarded-Proto value.

# kubectl  exec -it nginx-ingress-ingress-nginx-controller-799b9ccd64-kvxxf -n myns sh
/etc/nginx $ cat nginx.conf|grep X-Forwarded-Proto
                        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
                        proxy_set_header X-Forwarded-Proto                    "";
.
.
.
.

What you expected to happen:
Expect to have the support for removing the nginx controller added X-Forwarded-* headers from the URLs.

If we manually exec into the nginx controller pod and change the "$pass_access_scheme" to "" for X-Forwarded-Proto, works as expected.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

/etc/nginx $ /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.6
  Build:         6a73aa3b05040a97ef8213675a16142a9c95952a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

/etc/nginx $

Kubernetes version (use kubectl version):

# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:21:34Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.8", GitCommit:"66fee42707cd7f5a89f1987f7cb81b02dd19161c", GitTreeState:"clean", BuildDate:"2023-11-15T16:50:09Z", GoVersion:"go1.20.11", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Kubeadm based setup

• OS (e.g. from /etc/os-release): Linux

• Kernel (e.g. uname -a): Oracle Linux Server 7.8

• Install tools: kubeadm

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

    # helm ls -A | grep -i ingress
    

nginx-ingress myns 1 2024-02-28 22:00:50.671827003 -0800 PST deployed ingress-nginx-4.9.1 1.9.6
```

• If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

  # helm get values -n myns nginx-ingress
  USER-SUPPLIED VALUES:
  controller:
    admissionWebhooks:
      enabled: false
    allowSnippetAnnotation: true
    proxySetHeaders:
      X-Forwarded-Proto: ""
  

• If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

• if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses

     # kubectl describe ingressclasses
      Name:         nginx
     Labels:       app.kubernetes.io/component=controller
               app.kubernetes.io/instance=nginx-ingress
               app.kubernetes.io/managed-by=Helm
               app.kubernetes.io/name=ingress-nginx
               app.kubernetes.io/part-of=ingress-nginx
               app.kubernetes.io/version=1.9.6
               helm.sh/chart=ingress-nginx-4.9.1
     Annotations:  meta.helm.sh/release-name: nginx-ingress
                   meta.helm.sh/release-namespace: myns
     Controller:   k8s.io/ingress-nginx
     Events:       <none>
    

  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

[longwuyuan]

/remove-kind bug

• show the output of "kubectl -n ingress-nginx get cm ingress-nginx-controller -o yaml"
• I wonder if that header can be managed by https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-set-headers
  

[ashageetha]

# kubectl  -n myns get cm nginx-ingress-ingress-nginx-controller -o yaml
apiVersion: v1
data:
  allow-snippet-annotations: "false"
  proxy-set-headers: myns/nginx-ingress-ingress-nginx-custom-proxy-headers
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: nginx-ingress
    meta.helm.sh/release-namespace: myns
  creationTimestamp: "2024-02-29T06:00:51Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.9.6
    helm.sh/chart: ingress-nginx-4.9.1
  name: nginx-ingress-ingress-nginx-controller
  namespace: myns
  resourceVersion: "11272346"
  uid: bae7c8c0-ac7f-474f-9dff-dce93af6661a
  
  
# kubectl -n myns get cm nginx-ingress-ingress-nginx-custom-proxy-headers -o yaml
apiVersion: v1
data:
  X-Forwarded-Proto: ""
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: nginx-ingress
    meta.helm.sh/release-namespace: myns
  creationTimestamp: "2024-02-29T06:00:51Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.9.6
    helm.sh/chart: ingress-nginx-4.9.1
  name: nginx-ingress-ingress-nginx-custom-proxy-headers
  namespace: myns
  resourceVersion: "11272345"
  uid: 1fe1c0e2-d431-4898-9055-e88575f50f78

[longwuyuan]

What I tested and found out is that the configMap is for adding new and customized headers but not for changing the default headers.

[longwuyuan]

I have very little know on this topic but I found that this header is actually a factual info that is sent to the upstream (backend pod) . https://www.nginx.com/resources/wiki/start/topics/examples/forwarded/#using-the-forwarded-header

So in that context, why would you want to set this manually ? Its does not seem plausible that you make a HTTPS request to the cluster but you want to tell the backend pod that its GRPC or anything other than HTTPS !

To the layman reader here, please explain what is the real problem you want to solve by setting "X-Forwarded-Proto" header manually . Thanks

[longwuyuan]

/triage needs-information

[ashageetha]

One of our product URL is behaving different and causing too many redirects (recursively redirecting) if X-Forwarded-Proto is available in the request URL. Hence, we tried to remove this header and not able to achieve with NGINX. But we could modify and set accordingly with Traefik based ingress controller using customRequestHeaders.

[longwuyuan]

• I think we need a nginx expert to comment
• If you meant you tried with vanilla non-kubernetes nginx and it wa not possible, then highly likely that not possible in ingress-nginx controller as well
• Will keep google searching if nginx vanilla non-kubernetes was done or documented by someone

[longwuyuan]

After reading this https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto I am convinced you can not change it, in the context of saying your browser connects with HTTPS scheme but you change it manually to HTTP. If traefik changes it, I wonder how it impacts upstream getting false info. But like I said I am not a expert so wait for someone to comment