System Monitoring with ELK Stack: 14 Essential Configurations

The ELK (Elasticsearch, Logstash, Kibana) stack is a popular and powerful toolset for system monitoring and logging. In this article, we’ll explore 14 essential configurations that will help you get the most out of your ELK stack implementation.

What is ELK Stack?

The ELK stack is an open-source software solution for collecting, storing, and analyzing log data from various sources. It consists of three main components:

• Elasticsearch: A NoSQL database that stores and indexes log data.
• Logstash: A data processing pipeline that collects, filters, and transforms log data before sending it to Elasticsearch.
• Kibana: A web-based interface for visualizing and exploring log data stored in Elasticsearch.

Configuration 1: Collecting System Logs

Using Logstash

To collect system logs from various sources, you can use the following configuration:

“`json
input {
file {
path => “/var/log/syslog”
type => “syslog”
}
}

filter {
grok {
match => [“message”, “%{SYSLOG5424}”]
break_on_match => false
}
}

output {
elasticsearch {
hosts => [“localhost:9200”]
index => “system_logs”
}
}
“`

This configuration collects logs from /var/log/syslog and sends them to Elasticsearch for indexing.

Configuration 2: Monitoring CPU Usage

Using Kibana

To monitor CPU usage, you can create a dashboard in Kibana using the following visualization:

• Metric: cpu_percent
• Aggregate: Average
• Interval: 1 minute
• Range: Last 24 hours

This configuration will display the average CPU usage over the last 24 hours.

Configuration 3: Collecting Network Logs

Using Logstash

To collect network logs, you can use the following configuration:

“`json
input {
udp {
port => “514”
type => “syslog”
}
}

filter {
grok {
match => [“message”, “%{SYSLOG5424}”]
break_on_match => false
}
}

output {
elasticsearch {
hosts => [“localhost:9200”]
index => “network_logs”
}
}
“`

This configuration collects logs from UDP port 514 and sends them to Elasticsearch for indexing.

Configuration 4: Monitoring Memory Usage

Using Kibana

To monitor memory usage, you can create a dashboard in Kibana using the following visualization:

• Metric: mem_percent
• Aggregate: Average
• Interval: 1 minute
• Range: Last 24 hours

This configuration will display the average memory usage over the last 24 hours.

Configuration 5: Collecting Logstash Logs

Using Filebeat

To collect Logstash logs, you can use the following configuration:

“`json
input {
file {
path => “/var/log/logstash.log”
type => “logstash”
}
}

output {
elasticsearch {
hosts => [“localhost:9200”]
index => “logstash_logs”
}
}
“`

This configuration collects logs from /var/log/logstash.log and sends them to Elasticsearch for indexing.

Configuration 6: Monitoring Disk Space

Using Kibana

To monitor disk space, you can create a dashboard in Kibana using the following visualization:

• Metric: free_percent
• Aggregate: Average
• Interval: 1 minute
• Range: Last 24 hours

This configuration will display the average free disk space over the last 24 hours.

Configuration 7: Collecting HTTP Logs

Using Logstash

To collect HTTP logs, you can use the following configuration:

“`json
input {
http {
host => “localhost”
port => “8080”
type => “http”
}
}

filter {
grok {
match => [“message”, “%{HTTPDATE:timestamp} %{WORD:http_method} %{URIPATH:uri}”]
break_on_match => false
}
}

output {
elasticsearch {
hosts => [“localhost:9200”]
index => “http_logs”
}
}
“`

This configuration collects HTTP logs from https://localhost:8080 and sends them to Elasticsearch for indexing.

Configuration 8: Monitoring User Sessions

Using Kibana

To monitor user sessions, you can create a dashboard in Kibana using the following visualization:

• Metric: session_count
• Aggregate: Count
• Interval: 1 minute
• Range: Last 24 hours

This configuration will display the total number of user sessions over the last 24 hours.

Configuration 9: Collecting System Metrics

Using Logstash

To collect system metrics, you can use the following configuration:

“`json
input {
statsd {
host => “localhost”
port => 8125
type => “statsd”
}
}

output {
elasticsearch {
hosts => [“localhost:9200”]
index => “system_metrics”
}
}
“`

This configuration collects system metrics from statsd://localhost:8125 and sends them to Elasticsearch for indexing.

Configuration 10: Monitoring Database Queries

Using Kibana

To monitor database queries, you can create a dashboard in Kibana using the following visualization:

• Metric: query_count
• Aggregate: Count
• Interval: 1 minute
• Range: Last 24 hours

This configuration will display the total number of database queries over the last 24 hours.

Configuration 11: Collecting Security Logs

Using Logstash

To collect security logs, you can use the following configuration:

“`json
input {
file {
path => “/var/log/secure”
type => “security”
}
}

filter {
grok {
match => [“message”, “%{SYSLOG5424}”]
break_on_match => false
}
}

output {
elasticsearch {
hosts => [“localhost:9200”]
index => “security_logs”
}
}
“`

This configuration collects logs from /var/log/secure and sends them to Elasticsearch for indexing.

Configuration 12: Monitoring Application Performance

Using Kibana

To monitor application performance, you can create a dashboard in Kibana using the following visualization:

• Metric: response_time
• Aggregate: Average
• Interval: 1 minute
• Range: Last 24 hours

This configuration will display the average response time over the last 24 hours.

Configuration 13: Collecting Network Traffic

Using Logstash

To collect network traffic, you can use the following configuration:

“`json
input {
netflow {
host => “localhost”
port => 2055
type => “netflow”
}
}

output {
elasticsearch {
hosts => [“localhost:9200”]
index => “network_traffic”
}
}
“`

This configuration collects network traffic from netflow://localhost:2055 and sends it to Elasticsearch for indexing.

Configuration 14: Monitoring System Events

Using Kibana

To monitor system events, you can create a dashboard in Kibana using the following visualization:

• Metric: event_count
• Aggregate: Count
• Interval: 1 minute
• Range: Last 24 hours

This configuration will display the total number of system events over the last 24 hours.

In conclusion, these 14 ELK stack configurations will help you get started with collecting and monitoring various aspects of your system’s performance and activity. Remember to adjust the configurations according to your specific needs and requirements. Happy monitoring!

About the Author

Paul

Administrator

Visit Website View All Posts