check connection to broker is successful or not

[aliartiza75]

Description

Hello everyone,

Currently, what happens is that when the connection to Kafka broker is unsuccessful then logs of connection failure is printed on the screen.

I just wanna know is there any method that I can use to check the status of connection with kafka broker.

Currently, I am using AdminClient's list_topics method to check the connection with kafka broker.

Thanks in advance

Environment Details:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): confluent-kafka==0.11.6
• Apache Kafka broker version: kafka_2.11-1.0.0
• Client configuration: {'bootstrap.server': "localhost:9092"}
• Operating system: Ubuntu 18.04.1 LTS
• client logs

0 out-queue, 0 partially-sent requests
%4|1549975715.917|REQTMOUT|rdkafka#producer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Timed out 0 in-flight, 1 retry-queued, 0 out-queue, 0 partially-sent requests
%4|1549975721.921|REQTMOUT|rdkafka#producer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Timed out 0 in-flight, 1 retry-queued, 0 out-queue, 0 partially-sent requests

)

[rnpridgeon]

There is not although this should be of little concern to the application itself. Prior to establishing a successful connection with the broker(s) messages will be queued in the unassigned partition queue. Once the connection is established and the metadata response is received these messages will be moved from the unassigned queue to the appropriate partition queue and eventually transmitted to the leaders of these partitions. You also don't need to worry about retries or reconnects as these are also handled by librdkafka client. In the event librdkafka is unable to contact any of the brokers a _ALL_BROKERS_DOWN err will be returned to the err_cb. If this goes on for too long you will start to see _MSG_TIMED_OUT errors in your delivery reports as well. This means that retries have been exhausted and you'll need to decide what you want to do with the message contents at that point.

[rnpridgeon]

The following thread is highly relevant to your ask as well. I recommend giving it a quick once over as well #532

[aliartiza75]

@rnpridgeon thank you

[aliartiza75]

@rnpridgeon I have a few questions:

1. how long does it take to receive _MSG_TIMED_OUT error after the _ALL_BROKERS_DOWN error is received in err_cb?

2. what is the limit of the unassigned queue?

[rnpridgeon]

@aliartiza75 ,

There are a few situations in which librdkafka will propagate the _ALL_BROKERS_DOWN error almost immediately:

1. Connection Refused
2. Invalid addresses

There are times however where you may reach out to the wrong host and they do not actively refuse the connection. In this case it will continue to try and connect up until the system defined connection timeout.

After bootstrapping however if you lose connectivity to all brokers you can expect it to take roughly N brokers * (1 second + reconnect.backoff.ms).

You do however have application level control over when a _MSG_TIMED_OUT error is raised. To do so simply set the message.timeout.ms property to whatever value is appropriate for your application. For a complete list of configuration properties and their defaults can be found at the link below.

https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

The producer queue buffer size is also configurable and can be tuned using the queue.buffering.max.* properties from the link above.

[aliartiza75]

@rnpridgeon thank you