Monitoring and Alerting are one of the important aspects of Log Analytics. It helps you monitor your application and also alerts you through differenet channels like email, slack, Amazon chime etc about any issues proactively. In my previous posts I have explained about ELK stack installation and how to search logs or data using Elasticsearch queries. Now, lets see how we can monitor our data with different ways and also setup alerts. We will use Open Distro for Elasticsearch which is an Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise security, alerting, SQL, and more. So, lets get started.
InstallationLets first install Open Distro version of elasticsearch which provides out of the box alerting and monitoring features.
Open Distro ElasticsearchPlease follow below steps to download and install Elasticsearch
- Download the ZIP file.
- Extract the file to a directory, and open that directory at the command prompt.
- Run Open Distro for Elasticsearch:
curl -XGET https://localhost:9200 -u "admin:admin" --insecure curl -XGET https://localhost:9200/_cat/plugins?v -u "admin:admin" --insecure
Open Distro KibanaPlease follow below steps to download and install Kibana
- Download the ZIP.
- Extract the ZIP file to a directory and open that directory at the command prompt.
- If desired, modify
- Run Kibana:
Add Sample DataBefore we start, we need to have some data in elasticsearch to play with. So, lets go ahead and add some sample data. Follow below steps to add sample data Click on sample data
Click on Add Data That’s it. You have your sample data now.
Setup Alert Type (Destination)Now, lets first configure our alert type ie destination. Open Distro provides below options for destinations
- Amazon chime
- Custom Webhook
Create MonitorMonitor is a job that runs on a defined schedule and queries Elasticsearch. The results of these queries are then used as input for one or more triggers. Follow below steps to create monitor.Now, we will see how to create Monitor. You can setup these monitors as per your requirements. For example, if you want to monitor your critical APIs for its failure, you can setup monitor for the same. Another example could be, higher response times, higher error rate etc. Go to Alerting Click on Monitor Click on Create Monitor Click on Create You have successfully created Monitor, now lets go ahead and create Trigger for the same
Create TriggerTrigger is a conditions that, if met, generate alerts. Lets, create a Trigger to send email notification. Follow below steps to create Trigger Go go Alerting Click on Monitor Click on Monitor which we created in previous step Click on Create to create Trigger Define Trigger Configure Actions Click on Create You have successfully created trigger.
Alerting DashboardAlerting dashboard displays real time view of different alerts present in the system. You can acknowledge these alerts from dashboard.
RecommendationIf you want to go in depth and learn more about ELK stack I would recommend below books.
- Learning Elastic Stack 7.0: Distributed search, analytics, and visualization using Elasticsearch, Logstash, Beats, and Kibana, 2nd Edition
- ELK Stack A Complete Guide – 2020 Edition
- Learning ELK Stack by Saurabh Chhajed