Add retry attempt at pipeline start up #1014
Description
When co-hosting Logstash and an elasticsearch coordinating node on the same host. When the Host's CPU becomes overbooked (a high sustained load) then Logstash will fail pipelines upon start up because Elasticsearch doesn't respond fast enough.
We would like to have the option for Logstash to retry pipelines before marking them as failed. These are pipelines that are failing because the Elasticsearch output section is not connecting, these are not pipelines that are failing due to a code error. The pipeline is likely being marked as failed because Elasticsearch isn't responding fast enough and it is failing the initial health check. This health check is failing due to the high CPU load which is temporary due to the logstash start up impact and allowing it to retry would allow it to likely work because the CPU load would have hopefully normalized and elasticsearch would be able to respond.
It appears adding a timeout => 120 or greater can help in some cases
Currently there isn't a config to enable retry in pipeline initialization -- this could be an optional benefit.