Discover Log Data With ELK

In my earlier blog post I have written about overall ELK stack, what it is, how to get start and other basic information. Now we will move forward on this same topic and dive little bit more and learn about how to query your organization data, how to find out different patterns in your data.

Basically, we will learn how to discover your organization’s log data with this open-source tool ELK. Also, we will see how to create visualizations once you discover your data and how to create dashboard out of it. Such visualizations and dashboards are very useful to quickly give you an overview about your application.

So, let’s get started!

ELK Installation

The first step would be to get your ELK setup up and running. Please refer my earlier post to quickly install and run your ELK stack.

Once you are done with this step let’s move forward.

Add Sample Data into Elasticsearch

Now, we will add some sample data into Elasticsearch so that we can play around with the data. Please follow below steps to add e-commerce data into Elasticsearch. You can even skip this step if you already have your data into Elasticsearch.

1. Login into your Kibana instance with URL: http://localhost:5601/   
2. Once you login you will see an option to “Add Data”, click on the link
3. Go to sample data tab
4. First option would be “Sample eCommerce orders”, click on “Add Data”
5. This will add some sample eCommerce order transactions in Elasticsearch

Discover Data with Kibana

Now that you have some data in your Elasticsearch we will see different ways to discover your data. Please note that I have added sample data of eCommerce as an example only. ELK can be used to index any kinds of data.

Now, go to Menu -> Kibana -> Discover. You will need to select your Index from the dropdown, this will load the data indexed with selected index.

On the top, you will see text box with “search”. This is the tool we are going to use to discover your data. There are 2 ways you can use this search option.

1. Kibana Query Language (KQL): The Kibana Query Language (KQL) is a simple syntax for filtering Elasticsearch data using free text search or field-based search. KQL is only used for filtering data, and has no role in sorting or aggregating the data. KQL is able to suggest field names, values, and operators as you type.
2. Lucene query syntax: Lucene query syntax is available to Kibana users who opt out of the Kibana Query Language. The main reason to use the Lucene query syntax in Kibana is for advanced Lucene features, such as regular expressions or fuzzy term matching. However, Lucene syntax is not able to search nested objects or scripted fields.

We will learn both of these options.

Kibana Query Language

KQL has a different set of features than the Lucene query syntax. KQL is able to query nested fields and scripted fields. KQL does not support regular expressions or searching with fuzzy terms. Let’s see how to use KQL to apply filters to your data. First make sure KQL is selected as syntax option.

We already have our eCommerce data into Elasticsearch, we will try to filter that data using Kibana Query Language.

Now, let’s say you want to filter your data to know the list of sold products with discount percentage is greater than 10, you can simply do it as shown below.

As you can see in the response you have got 9 such products. You can even further filter this data by adding logical, Boolean operators based on your requirements.

Here is the list of options you have with KQL:

Terms query

A terms query uses exact search terms. Spaces separate each search term, and only one term is required to match the document. Use quotation marks to indicate a phrase match.

To query using exact search terms, enter the field name followed by : and then the values separated by spaces:

http.response.status_code:400 401 404

For text fields, this will match any value regardless of order:

http.response.body.content.text:Product created

To query for an exact phrase, use quotation marks around the values:

http.response.body.content.text:"Product created"

Field names are not required by KQL. When a field name is not provided, terms will be matched by the default fields in your index settings. To search across fields:

"product created"

Boolean queries

KQL supports or, and, and not. By default, and has a higher precedence than or. To override the default precedence, group operators in parentheses. These operators can be upper or lower case.

To match documents where response is 200, extension is php, or both:

response:200 or extension:php

To match documents where response is 200 and extension is php:

response:200 and extension:php

To match documents where response is 200 or 404.

response:(200 or 404)

To match documents where response is 200 and extension is either php or css:

response:200 and (extension:php or extension:css)

To match documents where response is not 200:

not response:200

To match documents where response is 200 but extension is not php or css.

response:200 and not (extension:php or extension:css)

Range queries

KQL supports >, >=, <, and <= on numeric and date types.

account_number >= 100 and items_sold <= 200

Date range queries

Typically, Kibana’s time filter is sufficient for setting a time range, but in some cases you might need to search on dates. Include the date range in quotes.

@timestamp < "2021-01-02T21:55:59"
@timestamp < "2021-01"
@timestamp < "2021"

Exist queries

An exist query matches documents that contain any value for a field, in this case, response:

response:*

Existence is defined by Elasticsearch and includes all values, including empty text.

Wildcard queries

Wildcards queries can be used to search by a term prefix or to search multiple fields.

To match documents where machine.os starts with win, such as “windows 7” and “windows 10”:

machine.os:win*

To match multiple fields:

machine.os*:windows 10

This syntax is handy when you have text and keyword versions of a field. The query checks machine.os and machine.os.keyword for the term windows 10.

Nested field queries

A main consideration for querying nested fields is how to match parts of the nested query to the individual nested documents. You can:

• Match parts of the query to a single nested document only. This is what most users want when querying on a nested field.
• Match parts of the query to different nested documents. This is how a regular object field works. This query is generally less useful than matching to a single document.

In the following document, items is a nested field. Each document in the nested field contains a name, stock, and category.

{
  "grocery_name": "Elastic Eats",
  "items": [
    {
      "name": "banana",
      "stock": "12",
      "category": "fruit"
    },
    {
      "name": "peach",
      "stock": "10",
      "category": "fruit"
    },
    {
      "name": "carrot",
      "stock": "9",
      "category": "vegetable"
    },
    {
      "name": "broccoli",
      "stock": "5",
      "category": "vegetable"
    }
  ]
}

Match a single document

To match stores that have more than 10 bananas in stock:

items:{ name:banana and stock > 10 }

items is the nested path. Everything inside the curly braces (the nested group) must match a single nested document.

The following query does not return any matches because no single nested document has bananas with a stock of 9.

items:{ name:banana and stock:9 }

Match different documents

The following subqueries are in separate nested groups and can match different nested documents:

items:{ name:banana } and items:{ stock:9 }

name:banana matches the first document in the array and stock:9 matches the third document in the array.

Match single and different documents

To find a store with more than 10 bananas that also stocks vegetables:

items:{ name:banana and stock > 10 } and items:{ category:vegetable }

The first nested group (name:banana and stock > 10) must match a single document, but the category:vegetables subquery can match a different nested document because it is in a separate group.

Nested fields inside other nested fields

KQL supports nested fields inside other nested fields—you have to specify the full path. In this document, level1 and level2 are nested fields:

{
  "level1": [
    {
      "level2": [
        {
          "prop1": "foo",
          "prop2": "bar"
        },
        {
          "prop1": "baz",
          "prop2": "qux"
        }
      ]
    }
  ]
}

To match on a single nested document:

level1.level2:{ prop1:foo and prop2:bar }

Lucene query syntax

To use Lucene query syntax first change the syntax from search text box.

The main reason to use the Lucene query syntax in Kibana is for advanced Lucene features, such as regular expressions or fuzzy term matching. However, Lucene syntax is not able to search nested objects or scripted fields.

To perform a free text search, simply enter a text string. For example, if you’re searching web server logs, you could enter safari to search all fields:

safari

To search for a value in a specific field, prefix the value with the name of the field:

status:200

To search for a range of values, use the bracketed range syntax, [START_VALUE TO END_VALUE]. For example, to find entries that have 4xx status codes, you could enter status:[400 TO 499].

status:[400 TO 499]

For an open range, use a wildcard:

status:[400 TO *]

To specify more complex search criteria, use the boolean operators AND, OR, and NOT. For example, to find entries that have 4xx status codes and have an extension of php or html:

status:[400 TO 499] AND (extension:php OR extension:html)

Kibana Visualization

Now that we have learned how to discover data using KQL or Lucene we will move forward to know how to create visualizations.

Kibana visualizations are based on Elasticsearch queries. By using a series of Elasticsearch aggregations to extract and process your data, you can create charts that show you the trends, spikes, and dips you need to know about.

You can create visualizations from a search saved from Discover or start with a new search query.

To create a visualization:

1. Click on Visualize in the side navigation.
2. Click the Create new visualization button or the + button.
3. Choose the visualization type. For our demo purpose we will create Pie chart.
4. Specify a search query to retrieve the data for your visualization:
   • To enter new search criteria, select the index pattern for the indices that contain the data you want to visualize. This opens the visualization builder with a wildcard query that matches all of the documents in the selected indices.
   • To build a visualization from a saved search, click the name of the saved search you want to use. This opens the visualization builder and loads the selected query.When you build a visualization from a saved search, any subsequent modifications to the saved search are automatically reflected in the visualization. To disable automatic updates, you can disconnect a visualization from the saved search.
5. In the visualization builder, choose the metric aggregation for the visualization’s Y axis

 

 

As we have entered size 2 (as shown in above dia), our pie chart shows only 2 categories

Now, we will change the size to 10, it shows max 10 categories.

We will save this visualization with “sales by category”

Kibana Dashboard

A Kibana dashboard displays a collection of visualizations and searches. You can arrange, resize, and edit the dashboard content and then save the dashboard so you can share it.

1. In the side navigation, click Dashboard.
2. Click Create new dashboard.
3. Click Add.
4. Use Add Panels to add visualizations and saved searches to the dashboard. If you have a large number of visualizations, you can filter the lists.
5. When you’re finished adding and arranging the panels, go to the menu bar and click Save.
6. In Save Dashboard, enter a dashboard title and optionally a description.
7. To store the time period specified in the time filter, enable Store time with dashboard.
8. Click Save.

This is how your dashboard looks like

Recommendation

If you want to go in depth and learn more about ELK stack I would recommend below books.

• Learning Elastic Stack 7.0: Distributed search, analytics, and visualization using Elasticsearch, Logstash, Beats, and Kibana, 2nd Edition
• ELK Stack A Complete Guide – 2020 Edition
• Learning ELK Stack by Saurabh Chhajed

Conclusion

So, in this post we have seen in detail how to query your log data, different ways to query it, how to create visualizations and how to create Dashboards. Hope this has helped you.

References

elastic.co

Tags: Application monitoring using ELK stackELK monitoringhow to create Kibana dashboardhow to create visualization using ELKhow to query data using ELKhow to query data using KQLkibana query language guidequery data using lucene