System Monitoring with ELK Stack: 21 Essential Configurations
The Elastic Stack, commonly known as the ELK Stack, is a powerful toolset for system monitoring and log analysis. It’s composed of three primary components: Elasticsearch (storage), Logstash (log collection and processing), and Kibana (visualizations and dashboards). In this article, we’ll explore 21 essential configurations to get the most out of your ELK Stack setup.
Prerequisites
Before diving into the configurations, make sure you have:
- An ELK Stack instance (Elasticsearch, Logstash, and Kibana) up and running.
- Familiarity with Linux and basic knowledge of networking concepts.
Configurations 1-5: Basic System Monitoring
Configuration 1: CPU Usage
Monitor system-wide CPU usage using a custom dashboards in Kibana. Create a visualization that displays the average CPU usage over time, and set up alerts for high CPU utilization.
“`yaml
Filebeat config to collect CPU data
filebeat.inputs:
– type: cpu
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“cpu.percent”: {“type”: “float”}
}
}
“`
Configuration 2: Memory Usage
Monitor system-wide memory usage using a custom dashboard in Kibana. Create a visualization that displays the average memory usage over time, and set up alerts for low or high memory utilization.
“`yaml
Filebeat config to collect memory data
filebeat.inputs:
– type: mem
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“mem.percent”: {“type”: “float”}
}
}
“`
Configuration 3: Disk Space Monitoring
Monitor available disk space using a custom dashboard in Kibana. Create a visualization that displays the average available disk space over time, and set up alerts for low disk space.
“`yaml
Filebeat config to collect disk data
filebeat.inputs:
– type: disk
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“disk.available_percent”: {“type”: “float”}
}
}
“`
Configuration 4: Network Interface Monitoring
Monitor network interface usage using a custom dashboard in Kibana. Create a visualization that displays the average network traffic over time, and set up alerts for high network utilization.
“`yaml
Filebeat config to collect network data
filebeat.inputs:
– type: net
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“net.bytes_in”: {“type”: “long”},
“net.bytes_out”: {“type”: “long”}
}
}
“`
Configuration 5: System Uptime Monitoring
Monitor system uptime using a custom dashboard in Kibana. Create a visualization that displays the average uptime over time, and set up alerts for system downtime.
“`yaml
Filebeat config to collect uptime data
filebeat.inputs:
– type: uptime
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“uptime”: {“type”: “float”}
}
}
“`
Configurations 6-10: Log Monitoring
Configuration 6: Apache Logs
Monitor Apache logs using a custom dashboard in Kibana. Create a visualization that displays the average request count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Apache data
filebeat.inputs:
– type: log
enabled: true
paths:
– /var/log/apache2/access.log
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“log.level”: {“type”: “keyword”},
“message”: {“type”: “text”}
}
}
“`
Configuration 7: Nginx Logs
Monitor Nginx logs using a custom dashboard in Kibana. Create a visualization that displays the average request count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Nginx data
filebeat.inputs:
– type: log
enabled: true
paths:
– /var/log/nginx/access.log
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“log.level”: {“type”: “keyword”},
“message”: {“type”: “text”}
}
}
“`
Configuration 8: MySQL Logs
Monitor MySQL logs using a custom dashboard in Kibana. Create a visualization that displays the average query count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect MySQL data
filebeat.inputs:
– type: log
enabled: true
paths:
– /var/log/mysql/error.log
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“log.level”: {“type”: “keyword”},
“message”: {“type”: “text”}
}
}
“`
Configuration 9: Postgres Logs
Monitor Postgres logs using a custom dashboard in Kibana. Create a visualization that displays the average query count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Postgres data
filebeat.inputs:
– type: log
enabled: true
paths:
– /var/log/postgresql/error.log
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“log.level”: {“type”: “keyword”},
“message”: {“type”: “text”}
}
}
“`
Configuration 10: Redis Logs
Monitor Redis logs using a custom dashboard in Kibana. Create a visualization that displays the average connection count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Redis data
filebeat.inputs:
– type: log
enabled: true
paths:
– /var/log/redis.log
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“log.level”: {“type”: “keyword”},
“message”: {“type”: “text”}
}
}
“`
Configurations 11-15: Service Monitoring
Configuration 11: MySQL Performance Monitoring
Monitor MySQL performance using a custom dashboard in Kibana. Create a visualization that displays the average query count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect MySQL data
filebeat.inputs:
– type: perfmon
enabled: true
counters:
– mysql_query_count
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“query_count”: {“type”: “long”}
}
}
“`
Configuration 12: Apache Performance Monitoring
Monitor Apache performance using a custom dashboard in Kibana. Create a visualization that displays the average request count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Apache data
filebeat.inputs:
– type: perfmon
enabled: true
counters:
– apache_request_count
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“request_count”: {“type”: “long”}
}
}
“`
Configuration 13: Nginx Performance Monitoring**
Monitor Nginx performance using a custom dashboard in Kibana. Create a visualization that displays the average request count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Nginx data
filebeat.inputs:
– type: perfmon
enabled: true
counters:
– nginx_request_count
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“request_count”: {“type”: “long”}
}
}
“`
Configuration 14: MySQL Connection Monitoring**
Monitor MySQL connections using a custom dashboard in Kibana. Create a visualization that displays the average connection count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect MySQL data
filebeat.inputs:
– type: connmon
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“connection_count”: {“type”: “long”}
}
}
“`
Configuration 15: Apache Connection Monitoring**
Monitor Apache connections using a custom dashboard in Kibana. Create a visualization that displays the average connection count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Apache data
filebeat.inputs:
– type: connmon
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“connection_count”: {“type”: “long”}
}
}
“`
Configurations 16-20: Error Monitoring
Configuration 16: Apache Error Logging**
Monitor Apache errors using a custom dashboard in Kibana. Create a visualization that displays the average error count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Apache data
filebeat.inputs:
– type: logmon
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“error_count”: {“type”: “long”}
}
}
“`
Configuration 17: MySQL Error Logging**
Monitor MySQL errors using a custom dashboard in Kibana. Create a visualization that displays the average error count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect MySQL data
filebeat.inputs:
– type: logmon
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“error_count”: {“type”: “long”}
}
}
“`
Configuration 18: Nginx Error Logging**
Monitor Nginx errors using a custom dashboard in Kibana. Create a visualization that displays the average error count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Nginx data
filebeat.inputs:
– type: logmon
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“error_count”: {“type”: “long”}
}
}
“`
Configuration 19: Postgres Error Logging**
Monitor Postgres errors using a custom dashboard in Kibana. Create a visualization that displays the average error count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Postgres data
filebeat.inputs:
– type: logmon
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“error_count”: {“type”: “long”}
}
}
“`
Configuration 20: Redis Error Logging**
Monitor Redis errors using a custom dashboard in Kibana. Create a visualization that displays the average error count over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect Redis data
filebeat.inputs:
– type: logmon
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“error_count”: {“type”: “long”}
}
}
“`
Configurations 21-25: Infrastructure Monitoring
Configuration 21: CPU Utilization Monitoring**
Monitor CPU utilization using a custom dashboard in Kibana. Create a visualization that displays the average utilization over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect data from /proc/loadavg
filebeat.inputs:
– type: loadavg
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“loadavg”: {“type”: “float”}
}
}
“`
Configuration 22: Memory Utilization Monitoring**
Monitor memory utilization using a custom dashboard in Kibana. Create a visualization that displays the average utilization over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect data from /proc/meminfo
filebeat.inputs:
– type: meminfo
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“mempercent”: {“type”: “float”}
}
}
“`
Configuration 23: Disk Utilization Monitoring**
Monitor disk utilization using a custom dashboard in Kibana. Create a visualization that displays the average utilization over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect data from /proc/diskstats
filebeat.inputs:
– type: diskstats
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“dskread”: {“type”: “long”},
“dskwrite”: {“type”: “long”}
}
}
“`
Configuration 24: Network Utilization Monitoring**
Monitor network utilization using a custom dashboard in Kibana. Create a visualization that displays the average utilization over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect data from /proc/net/dev
filebeat.inputs:
– type: netdev
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“rxbytes”: {“type”: “long”},
“txbytes”: {“type”: “long”}
}
}
“`
Configuration 25: Storage Utilization Monitoring**
Monitor storage utilization using a custom dashboard in Kibana. Create a visualization that displays the average utilization over time, and set up alerts for high traffic or errors.
“`yaml
Filebeat config to collect data from /proc/diskstats
filebeat.inputs:
– type: diskstats
enabled: true
Elasticsearch index mapping
{
“properties”: {
“@timestamp”: {“type”: “date”},
“dskread”: {“type”: “long”},
“dskwrite”: {“type”: “long”}
}
}
“`
This is a comprehensive list of configurations that can be used to monitor various aspects of a system. By combining these configurations, you can create a complete monitoring setup that covers all the essential metrics.
About the Author
