-
Notifications
You must be signed in to change notification settings - Fork 880
Description
Description
Hello,
I need to produce to Kafka on OpenShift/OKD cluster deployed using https://strimzi.io.
The specific of this setup is that Kafka brokers are not accessible from outside of the cluster. Kafka can be accessed via route service only which is TLS encrypted.
I try to access Kafka using 1.5.3 version of the client on Windows 10 and Windows Server 2019.
I use information from this source https://blog.mimacom.com/strimzi-okd/ from "External Access of Kafka" chapter.
I have extracted ca.crt as described in mentioned article. The problem is that this certificate has public key only. No private key.
Java client and Kafka consumer and producer console scripts work fine with .JKS trusted store created with this exported certificate (as depicted in the article). But Kafka .NET client does not.
I use following setting:
<add key="bootstrap.servers" value="kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxx.com:443"/>
<add key="ssl.endpoint.identification.algorithm" value="none" />
<add key="security.protocol" value="SSL" />
<add key="ssl.certificate.location" value="ca.crt" />
<add key="ssl.key.password" value="password" />
I read these settings from the app.config into Dictionary<string,string> and feed it to ProducerBuilder. Works fine for all other setting (not SSL related). So I'm sure that settings are adopted by ProducerBuilder.
I get SSL handshake failed error.
%3|1608580696.853|FAIL|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxx.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 479ms in state CONNECT)
%3|1608580696.853|ERROR|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxx]: 1/1 brokers are down
Could you please answer below questions?
- Is private key mandatory to configure SSL?
- If not then what is right configuration to use public key only?
Thank you in advance.
Adding console output with debug=protocol,security,broker
%7|1608583003.853|OPENSSL|rdkafka#producer-1| [thrd:app]: librdkafka built with OpenSSL version 0x1000211f
%7|1608583004.287|CERTROOT|rdkafka#producer-1| [thrd:app]: 74/74 certificate(s) successfully added from Windows Certificate Root store
%7|1608583004.287|SSL|rdkafka#producer-1| [thrd:app]: Loading public key from file ca.crt
%7|1608583004.290|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1608583004.290|BROKER|rdkafka#producer-1| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Added new broker with NodeId -1
%7|1608583004.290|CONNECT|rdkafka#producer-1| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1608583004.290|BRKMAIN|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Enter main broker thread
%7|1608583004.290|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.5.3 (0x10503ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0x282)
%7|1608583004.290|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Received CONNECT op
%7|1608583004.291|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1608583004.291|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: broker in state TRY_CONNECT connecting
%7|1608583004.291|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1608583004.295|OPENSSL|rdkafka#producer-2| [thrd:app]: librdkafka built with OpenSSL version 0x1000211f
%7|1608583004.302|CERTROOT|rdkafka#producer-2| [thrd:app]: 74/74 certificate(s) successfully added from Windows Certificate Root store
%7|1608583004.304|SSL|rdkafka#producer-2| [thrd:app]: Loading public key from file ca.crt
%7|1608583004.306|BRKMAIN|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1608583004.306|BROKER|rdkafka#producer-2| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Added new broker with NodeId -1
%7|1608583004.307|CONNECT|rdkafka#producer-2| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1608583004.307|BRKMAIN|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Enter main broker thread
%7|1608583004.307|INIT|rdkafka#producer-2| [thrd:app]: librdkafka v1.5.3 (0x10503ff) rdkafka#producer-2 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0x282)
%7|1608583004.307|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Received CONNECT op
%7|1608583004.307|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1608583004.308|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: broker in state TRY_CONNECT connecting
%7|1608583004.308|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1608583004.338|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 1960
%7|1608583004.338|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 1484
%7|1608583004.575|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Connected to ipv4#111.111.111.111:443
%7|1608583004.586|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Connected to ipv4#111.111.111.111:443
%7|1608583004.815|FAIL|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 524ms in state CONNECT) (_SSL)
%3|1608583004.816|FAIL|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 524ms in state CONNECT)
%7|1608583004.817|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state CONNECT -> DOWN
%3|1608583004.817|ERROR|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: 1/1 brokers are down
%3|1608583004.817|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 524ms in state CONNECT)
Checklist
Please provide the following information:
- A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
- Confluent.Kafka nuget version.
- Apache Kafka version.
- Client configuration.
- Operating system.
- Provide logs (with "debug" : "..." as necessary in configuration).
- Provide broker log excerpts.
- Critical issue.