Mine data with Elasticsearch
The purpose of this demo is to show how to feed data into Elasticsearch from API calls, Fluent, Aircraft Delays, BitCoin price, and a desired Twitter hashtag for data analytics and then archive them to S3. This demo will run Fluentd, Elastisearch, Kibana, and the Minio S3 Server in a microservices architecture.
Prerequisites
- Docker for Mac or Windows.
- This Git Repo
- 3GB of RAM or greater for Docker
- (Optional) Twitter API credentials
Building the images
unzip elastic-demo-master.zip
cd elastic-demo-master
docker-compose build
If you want to use the Twitter app to mine data from Twitter, modify the twitter section of docker-compose.yml with
your developer API credentials.
Starting the containers
docker-compose up -d
Check the status of the containers
docker ps
You should see the following containers running:
ONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
20e819e3013d elasticsearchdemo_fluent "/bin/sh -c 'fluentd…" 32 seconds ago Up 57 seconds elasticsearchdemo_fluent_1
72c2dc2b46cd elasticsearchdemo_flight-delays "/bin/sh -c 'sleep 1…" 36 seconds ago Up 58 seconds elasticsearchdemo_flight-delays_1
f242786ac556 elasticsearchdemo_twitter "/bin/sh -c 'npm ins…" 36 seconds ago Up 59 seconds elasticsearchdemo_twitter_1
285b2513310f elasticsearchdemo_elasticsearch-slave1 "/bin/sh -c 'bash /s…" 36 seconds ago Up 59 seconds elasticsearchdemo_elasticsearch-slave1_1
adb148937fdd elasticsearchdemo_elasticsearch-master "/bin/sh -c 'bash /s…" 36 seconds ago Up 58 seconds 0.0.0.0:9200->9200/tcp elasticsearchdemo_elasticsearch-master_1
4dee1accb840 elasticsearchdemo_kibana "/bin/sh -c 'cd /kib…" 36 seconds ago Up 58 seconds 0.0.0.0:5601->5601/tcp elasticsearchdemo_kibana_1
273beef5e868 elasticsearchdemo_elasticsearch-slave2 "/bin/sh -c 'bash /s…" 36 seconds ago Up 59 seconds elasticsearchdemo_elasticsearch-slave2_1
ab3f09566590 elasticsearchdemo_btc "/bin/sh -c 'npm ins…" 36 seconds ago Up 57 seconds elasticsearchdemo_btc_1
2abb3f520561 elasticsearchdemo_minio "/bin/sh -c './minio…" 36 seconds ago Up 59 seconds 0.0.0.0:9000->9000/tcp elasticsearchdemo_minio_1
Login to the Minio web console to see the logs
- Goto http://127.0.0.1:9000 in your web browser
- Login with accessKey1 for the username and verySecretKey1 for the password
- After a few minutes, the Elasticsearch log files will start appearing there from Fluent.
Accessing Kibana
- Goto https://0.0.0.0:5601 in your web browser
- Click the create button
- Start analyzing data
The default index of "logstash" will show you the Elasticsearch logs.
To view Twitter traffic, change the index to twitter or go to Management->Index Patterns-> + and then add twitter. Uncheck 'Index contains time-based events'.
To view Flight delay data, change the index to flightdata or go to Management->Index Patterns-> + and then add flightdata. Uncheck 'Index contains time-based events'.
To view the BitCoin price stats, change the index to btc or go to Management->Index Patterns-> + and then add btc.
Stopping and Erasing the demo
The following commands will stop and delete all running containers.
docker-compose kill
docker-compose rm -f
To start the demo again, simply run:
docker-compose up -d