Producer queue not getting emptied

[saurabh1920]

Description

In producer config, default value for message.timeout.ms is 300000. After this time, producer queue should automatically get cleared, I believe so. Please correct me if I am wrong.

producer = confluent_kafka.Producer(**prod_config)
This leads to message loss when the local producer queue gets full.

For more info, please refer: #341 (comment)

[edenhill]

The producer will retry temporary failures up to message.timeout.ms and then fail the message with ERR__TIMED_OUT.
You still need to call poll() or flush() to get the delivery report though.

[edenhill]

And there is no message loss on queue full, your application will need to handle QueueFull, usually by poll():ing and then retrying.

[saurabh1920]

Thanks for responding @edenhill ,
In my case, Kafka broker was down for sometime. And the producer was not accepting messages even when kafka turns up (probably local queue was full). After restarting application, producer started accepting data. In between I faced some data loss (loss count is around "queue.buffering.max.messages": 100000).

Please suggest how to handle such scenario.

[edenhill]

You either buffer data at the source (through backpressure when you get QueueFull), or you use large producer buffers.
As the cluster comes back up the queued messages will begin to be transmitted.