Monitoring Stack

Monitoring Stack

Comprehensive observability stack for monitoring platform health and performance.

Overview

The monitoring stack provides:

  • Metrics collection and alerting (Prometheus)
  • Log aggregation (Loki)
  • Dashboard visualization (Grafana)
  • Alert routing and notification (Alertmanager)
  • Container metrics (cAdvisor)
  • Node-level metrics (Node Exporter)
  • Log shipping (Promtail)

Technology Stack

  • Grafana 10.4.3: Dashboard and visualization
  • Prometheus: Metrics database and scraper
  • Loki: Log aggregation and querying
  • Alertmanager: Alert routing and grouping
  • Promtail: Log collector for container logs
  • Node Exporter: System metrics collection
  • cAdvisor: Container metrics collection
  • Docker for containerization

Architecture

┌────────────────────────────────────────────┐
│          Application Containers            │
└────────┬─────────────────────────┬─────────┘
         │                         │
         ▼                         ▼
   ┌─────────────┐          ┌──────────────┐
   │  Promtail   │          │ cAdvisor     │
   └──────┬──────┘          └────────┬─────┘
          │                          │
          └──────────────┬───────────┘
                         ▼
                    ┌─────────────┐
                    │  Prometheus │
                    └──────┬──────┘
                           │
         ┌─────────────────┼─────────────────┐
         ▼                 ▼                 ▼
    ┌────────────┐  ┌──────────────┐  ┌─────────────┐
    │  Grafana   │  │ Alertmanager │  │    Loki     │
    └────────────┘  └──────────────┘  └─────────────┘
         │                 │
         └────────┬────────┘
                  ▼
          ┌───────────────┐
          │ Slack/Email   │
          │ Notifications │
          └───────────────┘

Services

Grafana (Port 3001)

Dashboard and visualization platform:

  • Data source: Prometheus
  • Keycloak OAuth2 authentication
  • Role-based dashboard access
  • Alert creation and management
  • Custom dashboard creation

Configuration:

GF_AUTH_GENERIC_OAUTH_ENABLED: 'true'
GF_AUTH_DISABLE_LOGIN_FORM: 'true'
GF_AUTH_GENERIC_OAUTH_NAME: Keycloak
GF_SERVER_ROOT_URL: https://grafana-dev.hef.tum.de

Prometheus (Port 9090)

Metrics collection and storage:

  • Scrapes container and node metrics
  • Stores time-series data
  • Evaluates alert rules
  • 15-day retention (default)

Configuration: prometheus/prometheus.yml

Loki (Port 3100)

Log aggregation:

  • Collects logs from Promtail
  • Queryable log database
  • Integration with Grafana
  • LogQL query language

Configuration: loki/loki-config.yml

Alertmanager (Port 9093)

Alert routing and notifications:

  • Routes alerts from Prometheus
  • Groups related alerts
  • Sends notifications to Slack, email, etc.
  • Manages alert silencing

Configuration: alertmanager/alertmanager.yml

Promtail

Log shipper:

  • Collects Docker container logs
  • Ships to Loki
  • Automatic service discovery
  • Label-based log filtering

Configuration: promtail/promtail-config.yml

Node Exporter (Port 9100)

System-level metrics:

  • CPU, memory, disk usage
  • Network interface metrics
  • System load and processes

cAdvisor (Port 8081)

Container metrics:

  • Per-container CPU and memory
  • Network I/O statistics
  • Filesystem usage

Network

All services operate on the monitoring Docker network:

networks:
  monitoring:
    name: monitoring

Services can be discovered via Docker DNS:

  • grafana-monitoring
  • prometheus
  • loki
  • alertmanager
  • promtail
  • node-exporter
  • cadvisor

Access

Dashboards

Default dashboards provided for:

  • System overview
  • Container metrics
  • Application performance
  • Service health

Alerting

Alert rules defined in: prometheus/alert_rules.yml

Common alerts:

  • High CPU usage
  • High memory usage
  • Service down
  • Disk space low
  • Node down

References

KeycloakAudit Trail