Monitoring and Alerting with ELK

alerting-monitoring-using-elk
Monitoring and Alerting are one of the important aspects of Log Analytics. It helps you monitor your application and also alerts you through differenet channels like email, slack, Amazon chime etc about any issues proactively. In my previous posts I have explained about ELK stack installation and how to search logs or data using Elasticsearch queries. Now, lets see how we can monitor our data with different ways and also setup alerts. We will use Open Distro for Elasticsearch which is an Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise security, alerting, SQL, and more. So, lets get started.

Installation

Lets first install Open Distro version of elasticsearch which provides out of the box alerting and monitoring features.

Open Distro Elasticsearch

Please follow below steps to download and install Elasticsearch
  1. Download the ZIP file.
  2. Extract the file to a directory, and open that directory at the command prompt.
  3. Run Open Distro for Elasticsearch:
    .\bin\elasticsearch.bat
    
Now, lets verify that elasticsearch is successfully installed or not. After you start Open Distro for Elasticsearch, open a new command prompt window. Then send requests to the server to verify that it is up and running:
curl -XGET https://localhost:9200 -u "admin:admin" --insecure
curl -XGET https://localhost:9200/_cat/plugins?v -u "admin:admin" --insecure

Open Distro Kibana

Please follow below steps to download and install Kibana
  1. Download the ZIP.
  2. Extract the ZIP file to a directory and open that directory at the command prompt.
  3. If desired, modify config/kibana.yml.
  4. Run Kibana:
    .\bin\kibana.bat
Now, go to browser and enter http://localhost:5601/kibana-dashboard This will open Kibana Dashboard, enter username and password as admin/admin. To build more powerful dashboard please refer this book

Add Sample Data

Before we start, we need to have some data in elasticsearch to play with. So, lets go ahead and add some sample data. Follow below steps to add sample data Click on sample data

add sample data

Click on Add Data That’s it. You have your sample data now.

Setup Alert Type (Destination)

Now, lets first configure our alert type ie destination. Open Distro provides below options for destinations
  1. Amazon chime
  2. Slack
  3. Email
  4. Custom Webhook
We will setup our first Email Alert. Please follow below steps: Go to Alerting from left menu elastticsearch alerting Click on Destination destinations Click on Add Destination add destination Select “Email” from type add email destination Setup senders in Manage senders manage email sender   Setup email groups manage email group   Create Destination. That’s all. You have successfully setup your first Email alert type in elasticsearch. In next section we will see how to use this alert to send email notification.

Create Monitor

Monitor is a job that runs on a defined schedule and queries Elasticsearch. The results of these queries are then used as input for one or more triggers. Follow below steps to create monitor.Now, we will see how to create Monitor. You can setup these monitors as per your requirements. For example, if you want to monitor your critical APIs for its failure, you can setup monitor for the same. Another example could be, higher response times, higher error rate etc. Go to Alerting Click on Monitor monitor elasticsearch Click on Create Monitor create monitor-1 create monitor-2 Click on Create You have successfully created Monitor, now lets go ahead and create Trigger for the same

Create Trigger

Trigger is a conditions that, if met, generate alerts. Lets, create a Trigger to send email notification. Follow below steps to create Trigger Go go Alerting Click on Monitor Click on Monitor which we created in previous step edit monitor Click on Create to create Trigger create trigger Define Trigger define trigger Configure Actions configure action Click on Create You have successfully created trigger.

Alerting Dashboard

Alerting dashboard displays real time view of different alerts present in the system. You can acknowledge these alerts from dashboard. alerting dashboard ELK

Recommendation

If you want to go in depth and learn more about ELK stack I would recommend below books.

Conclusion

So, in this article we saw how to setup alerts, monitors using Elasticsearch. Hope this step by step guide on setting up alerts, monitors for your application will help you.   Tags:

Leave a Reply

Your email address will not be published. Required fields are marked *