Introduction

Elastic Stack (AKA ELK) Docker Composition, preconfigured with Security, Monitoring, Tools such as ElastAlert for alerting and Curator.

Based on Official Elastic Docker Images

Stack Version: 7.5.1.

You can change Elastic Stack version by setting ELK_VERSION in .env file and rebuild your images. Any version >= 7.0.0 is compatible with this template.

Main Points ?

• Configured as Production Single Node Cluster. (With a multi-node option for experimenting).
• Security Enabled (under basic license).
• SSL Enabled for Transport Layer and Kibana.
• Use Docker-Compose and .env to configure your stack.
• Automated Script that initializes and persist Elasticsearch's Keystore and SSL Certifications.
• Curator with Crond preconfigured for Automated Scheduled tasks (e.g Snapshots to S3).
• Self-Monitoring Metrics Enabled.
• Prometheus Exporters for Stack Metrics.
• Filebeat instance for shipping Stack logs to Elasticsearch itself.
• ElastAlert preconfigured for Alerting.
• Embedded Container Healthchecks for Stack Images.

More points at comparison with deviantony/docker-elk

Requirements

• Docker 17.05 or higher
• Docker-Compose 3 or higher

Setup

1. Clone the Repository, or:

2. Initialize Elasticsearch Keystore and SSL Certificates

$ make setup

3. Start Elastic Stack

$ make elk
---- OR ----
$ docker-compose up -d

4. Visit Kibana at https://localhost:5601

Username: elastic Password: changeme (or ELASTIC_PASSWORD value in .env)

Modify .env file for your needs, most importantly ELASTIC_PASSWORD that setup your superuser elastic's password, ELASTICSEARCH_HEAP & LOGSTASH_HEAP for Elasticsearch & Logstash Heap Size and ELK_VERSION for, yk, Stack Version.

Notice that Kibana is configured to use HTTPS, so you'll need to write https:// before localhost:5601 in the browser.

Additional Commands

To Start Monitoring and Prometheus Exporters

$ make monitoring

To Start Tools (ElastAlert and Curator

$ make tools

To Start ELK, Tools and Monitoring

$ make all

To Start 2 Extra Elasticsearch nodes (for development only)

$ make nodes

To Rebuild Images

$ make build

Bring down the stack.

$ make down

Make sure to run make setup if you changed ELASTIC_PASSWORD and to restart the stack after changing anything in .env.

Notes

• Adding Two Extra Nodes to the cluster will make the cluster depending on them and won't start without them again.

• Makefile is a wrapper around Docker-Compose commands, use make help to know every command.

• Elasticsearch will save its data to a volume named elasticsearch-data

• Elasticsearch Keystore (that contains passwords and credentials) and SSL Certificate are generated in the ./secrets directory by the setup command.

• Linux Users must set the following configuration as root

sysctl -w vm.max_map_count=262144

By default, Virtual Memory is not enough.

Configuration

• Some Configuration are parameterized in the .env file.
  • ELASTIC_PASSWORD, user elastic's password (default: changeme pls).
  • ELK_VERSION Elastic Stack Version (default: 7.5.1)
  • ELASTICSEARCH_HEAP, how much Elasticsearch allocate from memory (default: 1GB -good for development only-)
  • LOGSTASH_HEAP, how much Logstash allocate from memory.
  • Other configurations which their such as cluster name, and node name, etc.
• Elasticsearch Configuration in elasticsearch.yml at ./elasticsearch/config.
• Logstash Configuration in logstash.yml at ./elasticsearch/config/logstash.yml.
• Logstash Pipeline in main.conf at ./elasticsearch/pipeline/main.conf.
• Kibana Configuration in kibana.yml at ./kibana/config.
• ElastAlert Configuration in ./tools/elastalert/config.
• ElastAlert Alert rules in ./tools/elastalert/rules, head to ElastAlert docs to lookup how to create alerts.
• Curator Actions at ./tools/curator/actions and ./tools/curator/crontab.

Setting Up Keystore

You can extend the Keystore generation script by adding keys to ./setup/keystore.sh script. (e.g Add S3 Snapshot Repository Credentials)

To Re-generate Keystore:

make keystore

Enable SSL on HTTP

By default, only Transport Layer has SSL Enabled, to enable SSL on HTTP layer, add the following lines to elasticsearch.yml

## - http
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12

⚠️ Enabling SSL on HTTP layer will require all clients that connect to Elasticsearch to configure SSL connection for HTTP, this includes all the current configured parts of the stack (e.g Logstash, Kibana, Curator, etc) plus any library/binding that connects to Elasticsearch from your application code.

Monitoring Cluster

Prometheus Exporters

If you started Prometheus Exporters using make monitoring command. Prometheus Exporters will expose metrics at the following ports., Prometheus Exporter, Port, Note, --------------------------, ----------, ------------------------------------------------, elasticsearch-exporter, 9114, -, logstash-exporter, 9304, -, cadvisor-exporter, 8080, - To Monitor Each Container stats and metrics., ### Self-Monitoring is Enabled

Head to Stack Monitoring tab in Kibana to see cluster metrics for all stack components.

In Production, cluster metrics should be shipped to another dedicated monitoring cluster.

Comparison

One of the most popular elk on docker repositories is the awesome deviantony/docker-elk.
Elastdocker differs from deviantony/docker-elk in the following points.

• Security enabled by default using Basic license, not Trial.

• Persisting data by default in a volume.

• Run in Production Mode (by enabling SSL on Transport Layer, and add initial master node settings).

• Persisting Generated Keystore, and create an extendable script that makes it easier to recreate it every-time the container is created.

• Parameterize credentials in .env instead of hardcoding elastich:changeme in every component config.

• Parameterize all other Config like Heap Size.

• Add recommended environment configurations as Ulimits and Swap disable to the docker-compose.

• Make it ready to be extended into a multinode cluster.

• Configuring the Self-Monitoring and the Filebeat agent that ship ELK logs to ELK itself. (as a step to shipping it to a monitoring cluster in the future).

• Configured tools and Prometheus Exporters.

• The Makefile that simplifies everything into some simple commands.

License

MIT License
Copyright (c) 2019 Sherif Abdel-Naby

Contribution

PR(s) are Open and Welcomed.