An anomaly is any unusual change in behavior. Anomalies in your time-series data can lead to valuable insights. For example, for IT infrastructure data, an anomaly in the memory or CPU usage metric might help you uncover early signs of a system failure. Discovering anomalies using conventional methods such as creating visualizations and dashboards can be challenging. You can set an alert based on a static threshold, but this requires prior domain knowledge and is not adaptive to data that exhibits organic growth or seasonal behavior. The anomaly detection feature automatically detects anomalies in your Elasticsearch data in near real-time using the Random Cut Forest (RCF) algorithm. RCF is an unsupervised machine learning algorithm that models a sketch of your incoming data stream to compute an anomaly grade and confidence score value for each incoming data point. These values are used to differentiate an anomaly from normal variations. You can pair the anomaly detection plugin with the alerting plugin to notify you as soon as an anomaly is detected. So, lets see how to use this Open Source Anomaly detection feature in Open Distro for Elasticsearch. Please visit my earlier post for installation of Open Distro Elasticsearch and related plugins.

Create a Detector

A detector is an individual anomaly detection task. You can create multiple detectors, and all the detectors can run simultaneously, with each analyzing data from different sources. Follow below steps to create a detector.

1. Login to Kibana http://localhost:5601/ and go to Anomaly Detector
2. Click on Create Detector
3. Enter the Name of the detector and a brief Description. Make sure the name that you enter is unique and descriptive enough to help you to identify the purpose of this detector.
4. For Data source, choose the index that you want to use as the data source. You can optionally use index patterns to choose multiple indices.
5. Choose the Timestamp field in your index.
6. For Data filter, you can optionally filter the index that you chose as the data source. From the Filter type menu, choose Visual filter, and then design your filter query by selecting Fields, Operator, and Value, or choose Custom Expression and add in your own JSON filter query.
7. For Detector operation settings, define the Detector interval to set the time interval at which the detector collects data.
8. Click on Create

You have successfully created a detector

Add features to your detector

In this case, a feature is the field in your index that you want to check for anomalies. A detector can discover anomalies across one or more features. You must choose an aggregation method for each feature: average(), count(), sum(), min(), or max(). The aggregation method determines what constitutes an anomaly. For example, if you choose min(), the detector focuses on finding anomalies based on the minimum values of your feature. If you choose average(), the detector finds anomalies based on the average values of your feature. Follow below steps to add features:

1. On the Model configuration page, enter the Feature name.
2. For Find anomalies based on, choose the method to find anomalies. For Field Value menu, choose the field and the aggregation method. Or choose Custom expression, and add in your own JSON aggregation query.
3. Click on save And Start

Observe the results

Choose the Anomaly results tab. You will have to wait for some time to see the anomaly results. If the detector interval is 10 minutes, the detector might take more than an hour to start, as it’s waiting for sufficient data to generate anomalies. A shorter interval means the model passes the shingle process more quickly and starts to generate the anomaly results sooner. Use the profile detector operation to make sure you check you have sufficient data points.

• The Live anomalies chart displays the live anomaly results for the last 60 intervals. For example, if the interval is set to 10, it shows the results for the last 600 minutes. This chart refreshes every 30 seconds.
• The Anomaly history chart plots the anomaly grade with the corresponding measure of confidence.
• The Feature breakdown graph plots the features based on the aggregation method. You can vary the date-time range of the detector.
• The Anomaly occurrence table shows the Start time, End time, Data confidence, and Anomaly grade for each anomaly detected.

Set up alerts

To create a monitor to send you notifications when any anomalies are detected, choose Set up alerts. You’re redirected to the Alerting, Add monitor page. For steps to create a monitor and set notifications based on your anomaly detector, see my earlier post to create a Monitor. If you stop or delete a detector, make sure to delete any monitors associated with the detector.

Adjust the model

To see all the configuration settings, choose the Detector configuration tab.

1. To make any changes to the detector configuration, or fine tune the time interval to minimize any false positives, in the Detector configuration section, choose Edit.
   • You need to stop the detector to change the detector configuration. In the pop-up box, confirm that you want to stop the detector and proceed.
2. To enable or disable features, in the Features section, choose Edit and adjust the feature settings as needed. After you make your changes, choose Save and start detector.
   • Choose between automatically starting the detector (recommended) or manually starting the detector at a later time.

 

 Analyze historical data

Analyzing historical data helps you get familiar with the anomaly detection plugin. You can also evaluate the performance of a detector with historical data to further fine-tune it. Follow below steps to analyze historical data

1. Choose Historical detectors and Create historical detector.
2. Enter the Name of the detector and a brief Description.
3. For Data source, choose the index that you want to use as the data source. You can optionally use index patterns to choose multiple indices.
4. For Time range, select a time range for historical analysis.
5. For Detector settings, choose to use settings of an existing detector. Or choose the Timestamp field in your index, add individual features to the detector, and set the detector interval.
6. You can choose to run the historical detector automatically after creating.
7. Choose Create.

 

Manage your detectors

Go to the Detector details page to change or delete your detectors.

1. To make changes to your detector, choose the detector name to open the detector details page.
2. Choose Actions, and then choose Edit detector.
   • You need to stop the detector to change the detector configuration. In the pop-up box, confirm that you want to stop the detector and proceed.
3. After making your changes, choose Save changes.
4. To delete your detector, choose Actions, and then choose Delete detector.
   • In the pop-up box, type delete to confirm and choose Delete.

Recommendation

If you want to go in depth and learn more about ELK stack I would recommend below books.

• Learning Elastic Stack 7.0: Distributed search, analytics, and visualization using Elasticsearch, Logstash, Beats, and Kibana, 2nd Edition
• ELK Stack A Complete Guide – 2020 Edition
• Learning ELK Stack by Saurabh Chhajed

Conclusion

So, in this article we saw how to use anomaly detection feature available in Open Distro for Elasticsearch. Tags: anomaly detection using elasticsearchanomaly detection using ELKmachine learning in ELKopen source anomaly detection using ELK