Sajith's Blogspot: October 2018

12 October 2018

Analyze Custom Logs Using ELK

Last Month I was searching for a topic to write a good article. Then I thought to write an article regarding ELK stack. This article will describe the practical usage of the ELK stack. Let’s assume there is a merchant registration system in the eCommerce platform. Then at the end of the month we have to show the number of merchant registrations and merchant details ( location, country, etc ). For the above scenario there are lots of options to solve the problem. The common approach is to write data to the database and using any reporting library to show the report. But Is this approach is extensible ??. That is the problem. If the new requirement comes there are a lot of changes. Here what will do is write data to log files and analyze the data using ELK.

For this example I have generated log as json format. I have used this to generate json data set. This is the sample data set entry.

{“id”:1,”first_name”:”Freeman”,”last_name”:”Jowers”,”email”:”fjowers0@mashable.com”,”gender”:”Male”,”ip_address”:”15.128.77.162",”latitude”:9.9004655,”longitude”:13.0544185,”date”:”2017–10–29T17:47:59Z”,”country”:”Nigeria”}

Logstash Configuration

First I will show the logstash configuration I have used for read custom json logs & describe section by section.

input {
 file{
 path => [“/data/sample-data.json”]
 type => “json”
 start_position => “beginning”
 }
}
filter {
 grok {
 match => [‘message’,’(?<body>\”id\”:.*\”country\”:\”[^”]+\”)’]
 add_field => [“json_body”,”{%{body}}”]
 }
 json {
 source => “json_body”
 remove_field => [“message”,”body”,”json_body”]
 }
 mutate{
 add_field => [“[geoip][location]”,”%{[latitude]}”]
 add_field => [“[geoip][location]”,”%{[longitude]}”]
 }
 mutate{
 convert => [“[geoip][location]”,”float”]
 }
}
output {
 stdout {
 codec => rubydebug
 }
 elasticsearch {
 hosts => [“localhost:9200”]
 }
}

In logstash configuration three config sections

Input
Filter
OutPut

Logstash configurations build on filter based . There are currently 200 plugins available . In above configuration Input section describe how the event source can connect to logstash. In here event coming from file.

Filter section describe how the event are manipulated. In my case I have used two plugins to manipulate data. grok plugin convert separate json data entry. And mutate plugin convert longitude and latitude data to geo_point data.

Output section describes where the data need to output. It can be stream processing system [kafka] or database or may be to another logstash input. In above scenario set output as elastic search engine.

Read Logs Using Logstash

1	./logstash -f config-file.conf

Read Full article

Pages

12 October 2018

Analyze Custom Logs Using ELK

Logstash Configuration

Read Logs Using Logstash