Configure retry count

[ov7a]

My configuration is the following
java app => log4j2 SocketAppender (tcp) => Logstash: tcp input, es output => elasticsearch

Scenario:
Elasticsearch cluster is red and do not accept indexing by some reason.

Current behavior:
Logstash infinitely retries to index log into elasticsearch. Log4j2 waits for it and the java app simply hangs.

Desired behavior:
Logstash should retry, say, 5 times and then just reply to log4j2 with error somehow. Log4j2 throws an error => it can be logged in file and further investigated.

Is there an option to limit retry count for es output plugin?

(I've asked this on discuss, but received no reply )

• Version: 5.6.10
• Operating System: Debian 8

[jaschaio]

I am encountering the same issue.

Lots of infinite retries that will never succeed as the request is simply wrong:

Feb 14 15:22:36 vps logstash[12574]: [2019-02-14T15:22:36,705][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.1:9200/_bulk"}
Feb 14 15:22:38 vps logstash[12574]: [2019-02-14T15:22:38,746][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:22:42 vps logstash[12574]: [2019-02-14T15:22:42,759][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.1:9200/_bulk"}
Feb 14 15:22:50 vps logstash[12574]: [2019-02-14T15:22:50,771][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:23:06 vps logstash[12574]: [2019-02-14T15:23:06,800][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:23:38 vps logstash[12574]: [2019-02-14T15:23:38,830][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:24:42 vps logstash[12574]: [2019-02-14T15:24:42,857][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:25:46 vps logstash[12574]: [2019-02-14T15:25:46,866][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.1:9200/_bulk"}
Feb 14 15:26:50 vps logstash[12574]: [2019-02-14T15:26:50,873][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.1:9200/_bulk"}
Feb 14 15:27:54 vps logstash[12574]: [2019-02-14T15:27:54,900][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:28:58 vps logstash[12574]: [2019-02-14T15:28:58,908][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.1:9200/_bulk"}
Feb 14 15:30:02 vps logstash[12574]: [2019-02-14T15:30:02,934][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:31:06 vps logstash[12574]: [2019-02-14T15:31:06,965][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:32:10 vps logstash[12574]: [2019-02-14T15:32:10,998][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:33:15 vps logstash[12574]: [2019-02-14T15:33:15,028][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:34:19 vps logstash[12574]: [2019-02-14T15:34:19,054][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:35:23 vps logstash[12574]: [2019-02-14T15:35:23,069][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.1:9200/_bulk"}
Feb 14 15:36:27 vps logstash[12574]: [2019-02-14T15:36:27,091][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:37:31 vps logstash[12574]: [2019-02-14T15:37:31,102][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.1:9200/_bulk"}
Feb 14 15:38:35 vps logstash[12574]: [2019-02-14T15:38:35,118][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:39:39 vps logstash[12574]: [2019-02-14T15:39:39,134][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:40:43 vps logstash[12574]: [2019-02-14T15:40:43,149][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:41:47 vps logstash[12574]: [2019-02-14T15:41:47,164][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:42:51 vps logstash[12574]: [2019-02-14T15:42:51,194][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:43:55 vps logstash[12574]: [2019-02-14T15:43:55,201][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.1:9200/_bulk"}
Feb 14 15:44:59 vps logstash[12574]: [2019-02-14T15:44:59,212][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.1:9200/_bulk"}
Feb 14 15:46:03 vps logstash[12574]: [2019-02-14T15:46:03,238][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}
Feb 14 15:47:07 vps logstash[12574]: [2019-02-14T15:47:07,254][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:48:11 vps logstash[12574]: [2019-02-14T15:48:11,273][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:49:15 vps logstash[12574]: [2019-02-14T15:49:15,288][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.2:9200/_bulk"}
Feb 14 15:50:19 vps logstash[12574]: [2019-02-14T15:50:19,314][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://107.5.1.3:9200/_bulk"}

[wolframhaussig]

I had a problem today where I accidentally misconfigured an Elasticsearch Output on our production LogStash which broke the pipeline. As soon as I saw the error I fixed the pipeline but LogStash would not reload the pipeline as it still tried to send the data to Elasticsearch. So I wanted to restart LogStash to force the reloading of the pipeline but LogStash would not even restart. At last, an admin had to kill the logstash service so I was able to restart the process.

The pipeline error was:

[2021-09-28T06:49:00,640][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"security_exception", "reason"=>"action [indices:admin/auto_create] is unauthorized for user [logstash_admin], this action is granted by the index privileges [auto_configure,create_index,manage,all]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"}})
[2021-09-28T06:49:00,646][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}

As I have configured both persistent queues and DQL I would be glad to have a config to limit the number of retries. Ideally, after this limit is reached the document would be sent to the DLQ.