Using ILM for Filebeat Indices Retention

 

In this post we will review how to configure the Filebeat to use ILM for the logs retention management.

Filebeat collects the log files records, and sends them directly to the ElasticSearch. ElasticSearch saves the log records in an index. As time passes, the index size increases, and unless handled, we will eventually run out of disk space.

To solve this issue, we use the ElasticSearch Index Lifecycle Manager, the ILM. 

Using ILM for Dummies

First, Filebeat is indexing the log messages into an alias, and not directly into the ElasticSearch index. The ElasticSearch route the alias into a new created index.

Once a specific threshold, such as time or size, is breached, ElasticSearch triggers a redirection of the alias to a new index.

Eventually, after another time threshold is breached, ElasticSearch deletes the old index.

Configuring the FileBeat

To configure this behavior, we update the filebeat.yml file with the ILM configuration:

setup.ilm:
  enabled: true
  policy_name: "filebeat"
  rollover_alias: "filebeat"
  pattern: "{now/d}-000001"
  policy_file: /etc/ilm.json
  overwrite: true

And we add the ilm.json to configure the ILM policy:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1h"
          }
        }
      },
      "delete": {
        "min_age": "4h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

In this example, we rollover (change the alias target index) after one hour, and delete the index after four hours.

Final Notes

In this example we have shown triggering ILM using a time period trigger, but ILM can be configured also by a size trigger. In addition, ILM can have more phases, in addition to hot and delete. See the ElasticSearch documentation for more details.

Useful info: To check the status ILM of each index, using the ElasticSearch ILM explain API.