Prometheus + Grafana: Monitor Your Servers Like a Pro

Architecture Overview

Applications → Exporters → Prometheus → Grafana → Dashboards

• Prometheus: Scrapes and stores time-series metrics
• Exporters: Expose metrics from targets (node, app, DB)
• Grafana: Visualizes metrics with rich dashboards

Quick Start with Docker Compose

docker-compose.yml

version: "3.8"

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
      - GF_USERS_ALLOW_SIGN_UP=false
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--path.rootfs=/rootfs"
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]

Start Everything

docker compose up -d

• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin / admin123)

Prometheus Query Language (PromQL)

Basic Queries

# CPU usage percentage
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Disk usage percentage
(1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100

# Network receive bytes per second
rate(node_network_receive_bytes_total[5m])

Useful Functions

Function	Purpose
rate()	Per-second rate of increase
irate()	Instant rate (spike detection)
avg_over_time()	Average over time range
histogram_quantile()	Percentile from histogram
sum()	Aggregate across labels
topk()	Top N values
changes()	Number of value changes

Alerting Rules

prometheus-alerts.yml

groups:
  - name: server-alerts
    rules:
      - alert: HighCPU
        expr: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% for 5 minutes"

      - alert: HighMemory
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"

      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"

Add Alerts to Prometheus

# prometheus.yml
rule_files:
  - "prometheus-alerts.yml"

Grafana Dashboard Setup

Add Prometheus as Data Source

1. Grafana → Settings → Data Sources → Add
2. Select Prometheus
3. URL: http://prometheus:9090
4. Save & Test

Import Community Dashboards

1. Grafana → + → Import
2. Enter dashboard ID:
   • 1860: Node Exporter Full (comprehensive server metrics)
   • 12006: Docker container monitoring
   • 11074: MySQL monitoring
3. Select Prometheus data source
4. Import

Create Custom Panel

1. Edit dashboard → + Add Panel
2. Query: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
3. Visualization: Gauge
4. Title: "CPU Usage %"
5. Set thresholds: green < 60, yellow < 80, red > 80

Application Metrics

Expose Metrics from Node.js

import { register, Counter, Histogram, Gauge } from "prom-client";

// Create metrics
const httpRequests = new Counter({
  name: "http_requests_total",
  help: "Total HTTP requests",
  labelNames: ["method", "path", "status"],
});

const httpRequestDuration = new Histogram({
  name: "http_request_duration_seconds",
  help: "Request duration in seconds",
  labelNames: ["method", "path"],
});

const activeConnections = new Gauge({
  name: "active_connections",
  help: "Number of active connections",
});

// Middleware
app.use((req, res, next) => {
  const end = httpRequestDuration.startTimer();
  activeConnections.inc();

  res.on("finish", () => {
    httpRequests.inc({
      method: req.method,
      path: req.path,
      status: res.statusCode,
    });
    end({ method: req.method, path: req.path });
    activeConnections.dec();
  });

  next();
});

// Metrics endpoint
app.get("/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});

Add to Prometheus

scrape_configs:
  - job_name: "my-app"
    static_configs:
      - targets: ["app:3000"]
    metrics_path: "/metrics"

Log Aggregation (Bonus)

Loki + Promtail

Add to docker-compose.yml:

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - loki_data:/loki

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - ./promtail.yml:/etc/promtail/promtail.yml
    command: -config.file=/etc/promtail/promtail.yml

volumes:
  loki_data:

Add Loki as a data source in Grafana for log correlation.

Production Checklist

1. Set retention — --storage.tsdb.retention.time=30d
2. Add authentication — reverse proxy with auth
3. SSL/TLS — use Let's Encrypt with nginx
4. Backup Grafana — provision dashboards as JSON files
5. Monitor Prometheus itself — scrape /metrics endpoint
6. Use recording rules — precompute expensive queries
7. Alert routing — connect Alertmanager to Slack/PagerDuty