System Monitoring with ELK Stack: 14 Essential Configurations
The ELK (Elasticsearch, Logstash, Kibana) stack is a popular and powerful toolset for system monitoring and logging. In this article, we’ll explore 14 essential configurations that will help you get the most out of your ELK stack implementation.
What is ELK Stack?
The ELK stack is an open-source software solution for collecting, storing, and analyzing log data from various sources. It consists of three main components:
- Elasticsearch: A NoSQL database that stores and indexes log data.
- Logstash: A data processing pipeline that collects, filters, and transforms log data before sending it to Elasticsearch.
- Kibana: A web-based interface for visualizing and exploring log data stored in Elasticsearch.
Configuration 1: Collecting System Logs
Using Logstash
To collect system logs from various sources, you can use the following configuration:
“`json
input {
file {
path => “/var/log/syslog”
type => “syslog”
}
}
filter {
grok {
match => [“message”, “%{SYSLOG5424}”]
break_on_match => false
}
}
output {
elasticsearch {
hosts => [“localhost:9200”]
index => “system_logs”
}
}
“`
This configuration collects logs from /var/log/syslog and sends them to Elasticsearch for indexing.
Configuration 2: Monitoring CPU Usage
Using Kibana
To monitor CPU usage, you can create a dashboard in Kibana using the following visualization:
- Metric: cpu_percent
- Aggregate: Average
- Interval: 1 minute
- Range: Last 24 hours
This configuration will display the average CPU usage over the last 24 hours.
Configuration 3: Collecting Network Logs
Using Logstash
To collect network logs, you can use the following configuration:
“`json
input {
udp {
port => “514”
type => “syslog”
}
}
filter {
grok {
match => [“message”, “%{SYSLOG5424}”]
break_on_match => false
}
}
output {
elasticsearch {
hosts => [“localhost:9200”]
index => “network_logs”
}
}
“`
This configuration collects logs from UDP port 514 and sends them to Elasticsearch for indexing.
Configuration 4: Monitoring Memory Usage
Using Kibana
To monitor memory usage, you can create a dashboard in Kibana using the following visualization:
- Metric: mem_percent
- Aggregate: Average
- Interval: 1 minute
- Range: Last 24 hours
This configuration will display the average memory usage over the last 24 hours.
Configuration 5: Collecting Logstash Logs
Using Filebeat
To collect Logstash logs, you can use the following configuration:
“`json
input {
file {
path => “/var/log/logstash.log”
type => “logstash”
}
}
output {
elasticsearch {
hosts => [“localhost:9200”]
index => “logstash_logs”
}
}
“`
This configuration collects logs from /var/log/logstash.log and sends them to Elasticsearch for indexing.
Configuration 6: Monitoring Disk Space
Using Kibana
To monitor disk space, you can create a dashboard in Kibana using the following visualization:
- Metric: free_percent
- Aggregate: Average
- Interval: 1 minute
- Range: Last 24 hours
This configuration will display the average free disk space over the last 24 hours.
Configuration 7: Collecting HTTP Logs
Using Logstash
To collect HTTP logs, you can use the following configuration:
“`json
input {
http {
host => “localhost”
port => “8080”
type => “http”
}
}
filter {
grok {
match => [“message”, “%{HTTPDATE:timestamp} %{WORD:http_method} %{URIPATH:uri}”]
break_on_match => false
}
}
output {
elasticsearch {
hosts => [“localhost:9200”]
index => “http_logs”
}
}
“`
This configuration collects HTTP logs from https://localhost:8080 and sends them to Elasticsearch for indexing.
Configuration 8: Monitoring User Sessions
Using Kibana
To monitor user sessions, you can create a dashboard in Kibana using the following visualization:
- Metric: session_count
- Aggregate: Count
- Interval: 1 minute
- Range: Last 24 hours
This configuration will display the total number of user sessions over the last 24 hours.
Configuration 9: Collecting System Metrics
Using Logstash
To collect system metrics, you can use the following configuration:
“`json
input {
statsd {
host => “localhost”
port => 8125
type => “statsd”
}
}
output {
elasticsearch {
hosts => [“localhost:9200”]
index => “system_metrics”
}
}
“`
This configuration collects system metrics from statsd://localhost:8125 and sends them to Elasticsearch for indexing.
Configuration 10: Monitoring Database Queries
Using Kibana
To monitor database queries, you can create a dashboard in Kibana using the following visualization:
- Metric: query_count
- Aggregate: Count
- Interval: 1 minute
- Range: Last 24 hours
This configuration will display the total number of database queries over the last 24 hours.
Configuration 11: Collecting Security Logs
Using Logstash
To collect security logs, you can use the following configuration:
“`json
input {
file {
path => “/var/log/secure”
type => “security”
}
}
filter {
grok {
match => [“message”, “%{SYSLOG5424}”]
break_on_match => false
}
}
output {
elasticsearch {
hosts => [“localhost:9200”]
index => “security_logs”
}
}
“`
This configuration collects logs from /var/log/secure and sends them to Elasticsearch for indexing.
Configuration 12: Monitoring Application Performance
Using Kibana
To monitor application performance, you can create a dashboard in Kibana using the following visualization:
- Metric: response_time
- Aggregate: Average
- Interval: 1 minute
- Range: Last 24 hours
This configuration will display the average response time over the last 24 hours.
Configuration 13: Collecting Network Traffic
Using Logstash
To collect network traffic, you can use the following configuration:
“`json
input {
netflow {
host => “localhost”
port => 2055
type => “netflow”
}
}
output {
elasticsearch {
hosts => [“localhost:9200”]
index => “network_traffic”
}
}
“`
This configuration collects network traffic from netflow://localhost:2055 and sends it to Elasticsearch for indexing.
Configuration 14: Monitoring System Events
Using Kibana
To monitor system events, you can create a dashboard in Kibana using the following visualization:
- Metric: event_count
- Aggregate: Count
- Interval: 1 minute
- Range: Last 24 hours
This configuration will display the total number of system events over the last 24 hours.
In conclusion, these 14 ELK stack configurations will help you get started with collecting and monitoring various aspects of your system’s performance and activity. Remember to adjust the configurations according to your specific needs and requirements. Happy monitoring!
About the Author
