Exposing GRPC over a non-standard TCP port uses TCP/TLS health check on backend service

[jonathonbattista]

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

Release: v0.34.1 Build: v20200715-ingress-nginx-2.11.0-8-gda5fa45e2 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.19.1

Kubernetes version:

v1.21.3-gke.2001

Environment:

• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release):
  NAME="Alpine Linux" ID=alpine VERSION_ID=3.11.6 PRETTY_NAME="Alpine Linux v3.11" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://bugs.alpinelinux.org/"
• Kernel:
  Linux ingress-nginx-internal-controller-57d96fbbcd-nzspj 5.4.120+ #1 SMP Tue Jun 22 14:53:20 PDT 2021 x86_64 Linux

What happened:

I am exposing a non-standard TCP port (not 443) on an GRPC Ingress via this guide: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

I have confirmed the Ingress is listening on the port, but I also see the Controller is doing a TCP/TLS health check to the backend service, which we dont want. We just want an unencrypted health check.

I only see this message in the log regarding the TCP healthcheck:
[10.130.0.17] [10/Sep/2021:17:44:23 +0000] TCP 200 183 269 0.001

I did a packet capture with tcpdump/wireshark and verified there is a TLS handshake.

Why is the health check using TLS and not unencrypted TCP? Is this configurable anywhere? The backend isnt even using TLS.

Here are the server error logs indicating that it wasn't expecting TLS TCP traffic:

INFO: Transport failed
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 1603010108010001040303c16e175ac9b1af596e1a0f2d43
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:103)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:306)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:239)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
Sep 10, 2021 6:07:37 PM io.grpc.netty.shaded.io.grpc.netty.NettyServerTransport notifyTerminated

What you expected to happen:

Since no resources (Service, ConfigMap, Ingress, etc) are configured to use TCP/TLS for the Service backend health check, it should just use an unencrypted TCP health check.

How to reproduce it:

1. Follow this guide to expose a TCP port on an Ingress: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
2. Do a packet capture and check for a TLS handshake (SYN, ACK, SYN-ACK, FIN)

/kind bug

[k8s-ci-robot]

@jonathonbattista: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

had a conversation over zoom. One suspicion is that its the GCE LB which is doing the healthcheck, and not the controller. user is checking.

@nehaLohia27 is also checking

/assign @nehaLohia27

/remove-kind bug
/kind support
/triage needs-information

[k8s-ci-robot]

@longwuyuan: GitHub didn't allow me to assign the following users: nehaLohia27.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

had a conversation over zoom. One suspicion is that its the GCE LB which is doing the healthcheck, and not the controller. user is checking.

@nehaLohia27 is also checking

/assign @nehaLohia27

/remove-kind bug
/kind support
/triage needs-information

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nehaLohia27]

/assign

[nehaLohia27]

@jonathonbattista I am trying to check this issue. will have some comments after some investigation .

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale