Description
Issue description
When we upgrade Elasticsearch to 8.x and use legacy template (template_api
=>legacy
), the plugin resolves the composable style template as a default template and send to legacy API, obviously it doesn't work (see the produced error below).
The discussion with team brought to the idea of using legacy template api with custom template only. However, if ILM (ilm_* settings) is enabled, the plugin crashes. When plugin tries to set ILM settings to provided custom template, the plugin aligns on Elasticsearch version (pseudo: template_settings = version < 8 ? template['settings']: template['template']['settings']
). Obviously, with custom flexible template it is hard to guess where ILM settings located.
Steps to reproduce:
Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
- set
template_api
tolegacy
- connect to any (including cloud) Elasticsearch where its version is 8.x+
- run Logstash and check the log
Provide logs (if relevant):
[ERROR] 2022-12-13 23:42:54.690 [Ruby-0-Thread-12: :1] elasticsearch - Failed to install template
{:message=>"Got response code '400' contacting Elasticsearch at URL
'https://my-es-host.com:9200/_template/logstashb'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError,
:backtrace=>["/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'",
"/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:412:in `template_put'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:41:in `install'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install_template'", "/grid01/apps/PIPELINES/logstash-
8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch.rb:583:in
`install_template'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch.rb:349:in `finish_register'", "/grid01/apps/PIPELINES/logstash-
8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch.rb:305:in
`block in register'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/plugin_mixins/elasticsearch/common.rb:167:in `block in after_successful_connection'"]}
Acceptance Criteria
- Legacy template API is used with only custom templates
- Plugin should not crash, instead provide a clear message/guide what to do next