Index Lifecycle Management support for Logstash

[robbavey]

With Index lifecycle management currently being added to Elasticsearch, support should be added to Logstash along similar lines to the support currently being implemented for Beats

The implementation will require new configuration options, a changed default Elasticsearch template, and a default ILM policy that will be installed if one is not already present.

Configuration Options

New Options
ilm_enabled [boolean] (optional) - flag for enabling ILM integration. False by default, set to true to turn on ILM integration.
ilm_write_alias [string] (required) - write alias used for indexing data. If the write alias doesn’t exist, Logstash will create it and map it to the relevant index/pattern. Consider making this a required field when ilm_enabled is set
ilm_pattern [string] (optional) - appends “000001” by default for new index creation, subsequent rollover indices will increment based on this pattern i.e. “000002”
ilm_policy [string] (optional) - ILM policy to use, if undefined the default policy will be used.

Existing Options
index [string] - if ILM is enabled, this config option becomes the new index pattern to be used on new index creation.

Example:

output {
  elasticsearch {
    hosts => [...]
    ilm_enabled => true
    ilm_write_alias => “logstash”
    index => “logstash”
    ilm_pattern => “000001”
    ilm_policy => “logstash”
  }
}

Change to Default ES Template

When ILM is enabled, the default Logstash ES template should contain the ILM integration info which will be used when ILM is enabled.

The ES template should be dynamically configurable based on defined configurations in the ES output - index could map to “index_patterns”, policy could map to “index.lifecycle.name”, and write_alias could map to “index.lifecycle.rollover_alias”.

PUT _template/logstash
{
  "index_patterns": ["logstash-*"],
  "settings": {
    "index.lifecycle.name": "logstash",
    "index.lifecycle.rollover_alias": "logstash"
  },
  "mappings": {
    "_doc": {
      ...
    }
  }
}

At startup, Logstash will check for the existence of the write alias:

HEAD logstash

If write alias doesn’t exist yet, then it will create the new index, and then create the respective write alias:

PUT logstash-000001
{
  "aliases": {
      "logstash":{
            "is_write_index": true
     }
  }
}

From here on all data is sent to the write_alias alias, the Elasticsearch ILM implementation will take care of the index rotation behind the scenes.

New Default ILM Policy

Logstash will install a new ILM policy if an already existing policy is not explicitly defined in the configuration.

The default policy will have a size limit of 25GB and a date limit of 30 days:

PUT _ilm/logstash
{
  "policy" : {
    "phases": {
      "hot" : {
        "actions" : {
          "rollover" : {
            "max_size" : "25gb",
            "max_age":"30d"
          }
        }
      }
    }
  }
}

Size limit - we generally recommend a 30GB maximum per shard, so 25GB allows us to be close to that recommended limit while building in buffering for when rollovers may be executed late.

Date limit - this is a backstop for users who may have low ingest throughput, which allows them to avoid delete-by-query in the future in the case of misconfiguration. With a default date limit, these users would be able to delete/archive data with monthly indices if desired.

[untergeek]

As the author of Curator, I can't plus one this enough.