TP11.2 - Monitoring with Prometheus & Grafana

Objectives of the Lab

In this lab, you will:

✅ Configure Prometheus to scrape metrics
✅ Create Grafana dashboards
✅ Visualize application metrics
✅ Set up alerts

Estimated duration: 1h30

Prerequisites: Docker installed, TP11.1 completed

Context

You will set up a complete monitoring stack for the e-commerce application from TP1.

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Spring Boot    │────▶│   Prometheus    │────▶│    Grafana      │
│  /actuator      │     │   (Collection)  │     │  (Visualization)│
│  /prometheus    │     │                 │     │                 │
└─────────────────┘     └─────────────────┘     └─────────────────┘

Part 1: Docker Compose

Project Structure

project/
├── src/
├── pom.xml
├── docker-compose.yml
├── prometheus/
│   └── prometheus.yml
└── grafana/
    └── provisioning/
        └── datasources/
            └── datasource.yml

docker-compose.yml

version: '3.8'

services:
  # Your Spring Boot application
  app:
    build: .
    ports:
      - "8080:8080"
    environment:
      - SPRING_PROFILES_ACTIVE=docker
    networks:
      - monitoring

  # Prometheus - Metrics collection
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    networks:
      - monitoring

  # Grafana - Visualization
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    depends_on:
      - prometheus
    networks:
      - monitoring

volumes:
  prometheus_data:
  grafana_data:

networks:
  monitoring:
    driver: bridge

Part 2: Prometheus Configuration

prometheus/prometheus.yml

global:
  scrape_interval: 15s      # Collection frequency
  evaluation_interval: 15s  # Rule evaluation frequency

scrape_configs:
  # Prometheus scrapes itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Spring Boot Application
  - job_name: 'spring-boot-app'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 10s
    static_configs:
      - targets: ['app:8080']  # 'app' = Docker service name
        labels:
          application: 'ecommerce-app'
          environment: 'development'

  # If you run the app locally (outside Docker)
  # - job_name: 'spring-boot-local'
  #   metrics_path: '/actuator/prometheus'
  #   static_configs:
  #     - targets: ['host.docker.internal:8080']

Part 3: Grafana Configuration

grafana/provisioning/datasources/datasource.yml

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true

Part 4: Launch the Stack

Start the Services

# Start the stack
docker-compose up -d

# Check the logs
docker-compose logs -f

# Verify everything is working
curl http://localhost:8080/actuator/prometheus  # App
curl http://localhost:9090/targets              # Prometheus
# Grafana: http://localhost:3000 (admin/admin)

Without Docker (Local App)

If you run the app outside Docker:

# prometheus.yml - Modify the target
- job_name: 'spring-boot-app'
  metrics_path: '/actuator/prometheus'
  static_configs:
    - targets: ['host.docker.internal:8080']

Part 5: Verify Prometheus

Access Prometheus

Open http://localhost:9090

Check the Targets

Go to Status → Targets

You should see:

spring-boot-app with status UP

PromQL Queries

Test these queries in the Graph tab:

# JVM memory used
jvm_memory_used_bytes{area="heap"}

# HTTP requests per second
rate(http_server_requests_seconds_count[5m])

# P99 latency
histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))

# Pending orders
business_orders_pending

# Products low stock
business_products_low_stock

Part 6: Create a Grafana Dashboard

Open http://localhost:3000
Login: admin / admin
Change the password

Create a Dashboard

Click on + → Dashboard
Click on Add visualization
Select Prometheus as data source

Panel 1: Requests/Second

Query:

sum(rate(http_server_requests_seconds_count{application="ecommerce-app"}[5m]))

Settings:

Title: "Requests per Second"
Type: Stat or Time series

Panel 2: P95 Latency

Query:

histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket{application="ecommerce-app"}[5m])) by (le))

Settings:

Title: "P95 Latency"
Unit: seconds (s)

Panel 3: JVM Memory

Query:

jvm_memory_used_bytes{area="heap", application="ecommerce-app"} / 1024 / 1024

Settings:

Title: "Heap Memory Used (MB)"
Unit: megabytes

Panel 4: Orders by Status

Queries:

business_orders_pending
business_orders_completed
business_orders_cancelled

Settings:

Title: "Orders by Status"
Type: Stat (with repeat for each query)

Panel 5: Low Stock Products

Query:

business_products_low_stock

Settings:

Title: "Products with Low Stock"
Type: Gauge
Thresholds: 0-5 green, 5-10 yellow, 10+ red

Panel 6: Total Revenue

Query:

business_revenue_total

Settings:

Title: "Total Revenue"
Unit: Currency ($)

Part 7: Dashboard JSON (Import)

You can import this pre-configured dashboard:

{
  "title": "E-Commerce Application",
  "panels": [
    {
      "title": "Requests/sec",
      "type": "stat",
      "targets": [
        {
          "expr": "sum(rate(http_server_requests_seconds_count[5m]))",
          "refId": "A"
        }
      ],
      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0}
    },
    {
      "title": "P95 Latency",
      "type": "gauge",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket[5m])) by (le))",
          "refId": "A"
        }
      ],
      "fieldConfig": {
        "defaults": {"unit": "s", "thresholds": {"steps": [{"value": 0, "color": "green"}, {"value": 0.5, "color": "yellow"}, {"value": 1, "color": "red"}]}}
      },
      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0}
    },
    {
      "title": "Heap Memory (MB)",
      "type": "timeseries",
      "targets": [
        {
          "expr": "jvm_memory_used_bytes{area=\"heap\"} / 1024 / 1024",
          "refId": "A"
        }
      ],
      "fieldConfig": {"defaults": {"unit": "decmbytes"}},
      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
    },
    {
      "title": "Orders Pending",
      "type": "stat",
      "targets": [{"expr": "business_orders_pending", "refId": "A"}],
      "gridPos": {"h": 4, "w": 4, "x": 0, "y": 4}
    },
    {
      "title": "Low Stock Products",
      "type": "stat",
      "targets": [{"expr": "business_products_low_stock", "refId": "A"}],
      "fieldConfig": {
        "defaults": {"thresholds": {"steps": [{"value": 0, "color": "green"}, {"value": 5, "color": "yellow"}, {"value": 10, "color": "red"}]}}
      },
      "gridPos": {"h": 4, "w": 4, "x": 4, "y": 4}
    },
    {
      "title": "Total Revenue",
      "type": "stat",
      "targets": [{"expr": "business_revenue_total", "refId": "A"}],
      "fieldConfig": {"defaults": {"unit": "currencyUSD"}},
      "gridPos": {"h": 4, "w": 4, "x": 8, "y": 4}
    }
  ],
  "refresh": "10s",
  "time": {"from": "now-1h", "to": "now"}
}

To import:

Dashboard → Import
Paste the JSON
Select Prometheus datasource
Import

Part 8: Generate Traffic

Test Script

#!/bin/bash
# generate-traffic.sh

API_URL="http://localhost:8080"

# Create orders
for i in {1..20}; do
  curl -X POST "$API_URL/api/orders" \
    -H "Content-Type: application/json" \
    -d "{\"customerEmail\":\"user$i@test.com\",\"total\":$((RANDOM % 500 + 100))}"
  sleep 0.5
done

# Browse products
for i in {1..50}; do
  curl "$API_URL/api/products" > /dev/null 2>&1
  sleep 0.2
done

echo "Traffic generated!"

Run the script and observe the metrics in Grafana.

Part 9: Alerts (Bonus)

Add Alerts in Grafana

Go to a panel (e.g., Low Stock Products)
Click on Alert → Create alert rule
Configure:
- Condition: query A > 10
- Evaluate every: 1m
- For: 5m (must be true for 5 min)
Add a notification (email, Slack, etc.)

Alerts in Prometheus

Create prometheus/alert.rules.yml:

groups:
  - name: spring-boot-alerts
    rules:
      - alert: HighMemoryUsage
        expr: jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"} > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage detected"
          description: "Heap memory is above 90%"

      - alert: HighErrorRate
        expr: sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) / sum(rate(http_server_requests_seconds_count[5m])) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate"
          description: "Error rate is above 5%"

      - alert: LowStock
        expr: business_products_low_stock > 10
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Many products with low stock"

Final Checklist

Docker Compose starts without error
Prometheus scrapes the application (Target UP)
Metrics visible in Prometheus
Functional Grafana dashboard
Business metrics (orders, products) displayed
Automatic dashboard refresh

🎯 Module Completed!

Congratulations! You have completed Module 11 - Spring Boot Actuator and Monitoring.

Summary:

Chapter 1: Introduction to Actuator
Chapter 2: Endpoints
Chapter 3: Micrometer Metrics
Chapter 4: Health Checks
Lab 1: Actuator Configuration
Lab 2: Prometheus & Grafana

Skills acquired:

✅ Complete Actuator configuration
✅ Custom health checks
✅ Business metrics with Micrometer
✅ Prometheus/Grafana monitoring stack
✅ Dashboards and visualizations

➡️ Back to Module 11

⬅️ Back to Course Outline

Objectives of the Lab​

Context​

Part 1: Docker Compose​

Project Structure​

docker-compose.yml​

Part 2: Prometheus Configuration​

prometheus/prometheus.yml​

Part 3: Grafana Configuration​

grafana/provisioning/datasources/datasource.yml​

Part 4: Launch the Stack​

Start the Services​

Without Docker (Local App)​

Part 5: Verify Prometheus​

Access Prometheus​

Check the Targets​

PromQL Queries​

Part 6: Create a Grafana Dashboard​

Login​

Create a Dashboard​

Panel 1: Requests/Second​

Panel 2: P95 Latency​

Panel 3: JVM Memory​

Panel 4: Orders by Status​

Panel 5: Low Stock Products​

Panel 6: Total Revenue​

Part 7: Dashboard JSON (Import)​

Part 8: Generate Traffic​

Test Script​

Part 9: Alerts (Bonus)​

Add Alerts in Grafana​

Alerts in Prometheus​

Final Checklist​

🎯 Module Completed!​

Objectives of the Lab

Context

Part 1: Docker Compose

Project Structure

docker-compose.yml

Part 2: Prometheus Configuration

prometheus/prometheus.yml

Part 3: Grafana Configuration

grafana/provisioning/datasources/datasource.yml

Part 4: Launch the Stack

Start the Services

Without Docker (Local App)

Part 5: Verify Prometheus

Access Prometheus

Check the Targets

PromQL Queries

Part 6: Create a Grafana Dashboard

Login

Create a Dashboard

Panel 1: Requests/Second

Panel 2: P95 Latency

Panel 3: JVM Memory

Panel 4: Orders by Status

Panel 5: Low Stock Products

Panel 6: Total Revenue

Part 7: Dashboard JSON (Import)

Part 8: Generate Traffic

Test Script

Part 9: Alerts (Bonus)

Add Alerts in Grafana

Alerts in Prometheus

Final Checklist

🎯 Module Completed!