Skip to content

Plugin fails to install default template when using legacy API with Elasticsearch 8.x (or higher) versions. #1108

Closed
@mashhurs

Description

@mashhurs

Issue description

When we upgrade Elasticsearch to 8.x and use legacy template (template_api=>legacy), the plugin resolves the composable style template as a default template and send to legacy API, obviously it doesn't work (see the produced error below).
The discussion with team brought to the idea of using legacy template api with custom template only. However, if ILM (ilm_* settings) is enabled, the plugin crashes. When plugin tries to set ILM settings to provided custom template, the plugin aligns on Elasticsearch version (pseudo: template_settings = version < 8 ? template['settings']: template['template']['settings']). Obviously, with custom flexible template it is hard to guess where ILM settings located.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

  1. set template_api to legacy
  2. connect to any (including cloud) Elasticsearch where its version is 8.x+
  3. run Logstash and check the log

Provide logs (if relevant):

[ERROR] 2022-12-13 23:42:54.690 [Ruby-0-Thread-12: :1] elasticsearch - Failed to install template 
{:message=>"Got response code '400' contacting Elasticsearch at URL 
'https://my-es-host.com:9200/_template/logstashb'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, 
:backtrace=>["/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", 
"/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:412:in `template_put'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:41:in `install'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install_template'", "/grid01/apps/PIPELINES/logstash-
8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch.rb:583:in 
`install_template'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch.rb:349:in `finish_register'", "/grid01/apps/PIPELINES/logstash-
8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch.rb:305:in 
`block in register'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/plugin_mixins/elasticsearch/common.rb:167:in `block in after_successful_connection'"]}

Acceptance Criteria

  • Legacy template API is used with only custom templates
  • Plugin should not crash, instead provide a clear message/guide what to do next

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions