Unable to catch python timeout error

[zaheer032]

Description

Unable to catch Kafka timeout error

How to reproduce

My confluent-kafka python- version 1.7.0 producer code is as below

from confluent_kafka import KafkaError, KafkaException

def publish_data():

   try:

        producer.produce(
            topic=topic,
            value={"produced_epoch": produced_epoch },
        )
        producer.flush()
    except (KafkaError, RuntimeError, KafkaException, Exception) as e:
        logging.error(f'Error publishing to kafka : {e}')
        return False
    return True

While producing the code I am getting the below error in Kafka broker

%5|1634047381.445|REQTMOUT|rdkafka#producer-1| [thrd:ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2]: ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2: Timed out ProduceRequest in flight (after 60844ms, timeout #0) %5|1634047381.445|REQTMOUT|rdkafka#producer-1| [thrd:ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2]: ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2: Timed out ProduceRequest in flight (after 60837ms, timeout #1) %5|1634047381.445|REQTMOUT|rdkafka#producer-1| [thrd:ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2]: ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2: Timed out ProduceRequest in flight (after 60816ms, timeout #2) %5|1634047381.445|REQTMOUT|rdkafka#producer-1| [thrd:ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2]: ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2: Timed out ProduceRequest in flight (after 60809ms, timeout #3) %5|1634047381.445|REQTMOUT|rdkafka#producer-1| [thrd:ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2]: ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2: Timed out ProduceRequest in flight (after 60788ms, timeout #4) %4|1634047381.445|REQTMOUT|rdkafka#producer-1| [thrd:ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2]: ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2: Timed out 31 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests %3|1634047381.445|FAIL|rdkafka#producer-1| [thrd:ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2]: ssl://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093/2: 31 request(s) timed out: disconnect (after 108069ms in state UP)

I want to catch the REQTMOUT error and return False when error. But unable to do the same. Can anyone help me with it.

Checklist

Please provide the following information:

• confluent-kafka-python : 1.7.0 with librdkafka
• Apache Kafka broker version:
• Client configuration: {...}
• Operating system: linux
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

[jliunyu]

Hi @zaheer032 , thanks for asking.

On_delivery callback will be a good idea to catch up the MSG_TIMED_OUT errors and in the event librdkafka is unable to contact any of the brokers a _ALL_BROKERS_DOWN err will be returned to the err_cb.

The answer You also don't need to worry about retries or reconnects as these are also handled by librdkafka client. In the event librdkafka is unable to contact any of the brokers a _ALL_BROKERS_DOWN err will be returned to the err_cb. If this goes on for too long you will start to see _MSG_TIMED_OUT errors in your delivery reports as well. This means that retries have been exhausted and you'll need to decide what you want to do with the message contents at that point. from #539 (comment) is still good for this one.

Example for error_cb: https://github.com/confluentinc/confluent-kafka-python/blob/master/tests/test_Producer.py#L10
Example for on_delivery callback: https://github.com/confluentinc/confluent-kafka-python/blob/master/tests/test_Producer.py#L29