Index Lifecycle Management Support

[robbavey]

• Add configuration options to plugin
• Implement basic Index Lifecycle Management support
• Create default policy, and add if specified
• Check gracefully handles oss with ILM and pre-ILM versions of Elasticsearch
• Add integration tests
• Add License enforcement
• Add docs . (will do this in a separate PR)
• Check todos, and clean up code

Resolves #798

[LucaWintergerst]

I was wondering if this would also support dynamic index names based on a field of an event, but from what Rob told me this is not possible right now:
ilm_write_alias => “%{foobar}”

How bad of an idea would it be to do something like the following to support it being dynamic?

my_index_name = event.sprintf(@ilm_index_name_field)
if !@@my_ilm_indices.include?(my_index_name)
  maybe_create_write_alias
  maybe_create_ilm_policy
  @@my_ilm_indices.push(my_index_name)
end

The following issues come to mind:

• logstash is stateless and would have to collect the list of known patterns over and over again after each restart. We could mitigate this by adding an initial list for known indices where we would add aliases and ILMs at startup
• it hurts the performance as we have to check this for every event
• the array could grow too large and break something. I suppose we could add soft limits to restrict it to a max size
• likely a whole lot more

Just wanted to bring it up as a lot of users rely on the dynamic index creation based on event data

[guyboertje]

LGTM

[robbavey]

Thanks @guyboertje!