Lab — Monitoring & Alerting

Difficulty: Intermediate · Estimated time: 45-60 min

Set up Prometheus and Grafana to monitor a service, trigger alerts during an outage, and visualize recovery.

---

🎯 Objective

• Deploy Prometheus & Grafana to your cluster
• Create an alert for high error rates
• Trigger the alert by breaking the app (from Incident Simulation lab)
• Watch the alert fire and resolve

---

🛠 Prerequisites

• Kubernetes cluster with kubectl access
• Helm installed (helm version)
• hello app deployed

---

📦 Step 1 — Install Prometheus & Grafana

We’ll use the kube-prometheus-stack Helm chart.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring

Check pods:

kubectl get pods -n monitoring

---

🔑 Step 2 — Access Grafana

Get admin password:

kubectl get secret monitoring-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 --decode

Port-forward Grafana:

kubectl port-forward svc/monitoring-grafana -n monitoring 3000:80

Access at: http://localhost:3000

Login: - User: admin - Pass: (password from above)

---

📈 Step 3 — Create a Dashboard

1. In Grafana, click + → Create Dashboard → Add new panel.
2. Query (PromQL):

   sum(rate(http_requests_total{status=~"5.."}[5m]))
   

3. Set Visualization to Time series.
4. Save as "Error Rate Dashboard".

---

🚨 Step 4 — Create an Alert Rule

Edit PrometheusRule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: hello-rules
  namespace: monitoring
spec:
  groups:
  - name: hello-alerts
    rules:
    - alert: HighErrorRate
      expr: sum(rate(http_requests_total{app="hello",status=~"5.."}[5m])) > 5
      for: 2m
      labels:
        severity: page
      annotations:
        summary: "High HTTP 5xx error rate on hello service"
        description: "Error rate >5 req/min for >2m."

Apply:

kubectl apply -f hello-alerts.yaml

---

🧨 Step 5 — Trigger the Alert

From the Incident Simulation Lab: 1. Deploy broken image:

kubectl set image deployment/hello hello=ghcr.io/<your-org>/hello:broken

2. Wait 2–3 minutes.

Check alert status:

kubectl port-forward svc/monitoring-kube-prometheus-alertmanager -n monitoring 9093:9093

Open http://localhost:9093 — you should see HighErrorRate firing.

---

🏁 Step 6 — Resolve

Rollback:

kubectl rollout undo deployment hello

Alert will transition to Resolved after error rate drops.

---

📬 Step 7 — (Optional) Integrate with Slack

1. Create Slack Incoming Webhook.
2. Create alertmanager-config.yaml:

   global:
     resolve_timeout: 5m
   receivers:
   - name: slack
     slack_configs:
     - api_url: "<SLACK_WEBHOOK_URL>"
       channel: "#alerts"
   route:
     receiver: slack
   

3. Apply:

   kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml=alertmanager-config.yaml -n monitoring --dry-run=client -o yaml | kubectl apply -f -
   kubectl rollout restart statefulset alertmanager-monitoring-kube-prometheus-alertmanager -n monitoring
   

---

✅ Success Criteria

• [ ] Grafana dashboard shows error rates
• [ ] Prometheus rule triggers when app fails
• [ ] Alert fires in Alertmanager
• [ ] Alert resolves after rollback
• [ ] (Optional) Slack receives alert

---

Monitoring without alerting is just a pretty graph. Alerting without context is just noise. The goal is actionable, timely insight.