Skip to content

Lab — Full SRE Drill

Difficulty: Advanced · Estimated time: 90-120 min

A complete hands-on simulation of an SRE shift, combining detection, triage, mitigation, resolution, and postmortem — all in one exercise.

🎯 Objective

  • Detect an outage via monitoring
  • Investigate using Linux & Kubernetes tools
  • Fix a broken container image
  • Redeploy the service to Kubernetes
  • Verify recovery in dashboards
  • Complete a postmortem

🛠 Lab Setup

Prerequisites

📦 Step 1 — Start Healthy State

  1. Ensure hello app is running version 1.0: 
    kubectl set image deployment/hello hello=ghcr.io/<your-org>/hello:1.0
kubectl rollout status deployment hello
  2. Confirm readiness: 
    kubectl get pods -l app=hello
curl http://<hello-service-ip>:8080

🚨 Step 2 — Trigger a Failure

Simulate a bad deploy: 

kubectl set image deployment/hello hello=ghcr.io/<your-org>/hello:broken
Expected: - Pods crashloop - Error rate rises - Alerts fire in Alertmanager

🔍 Step 3 — Detect & Triage

From Monitoring

  1. Open Grafana Error Rate Dashboard.
  2. See HighErrorRate alert firing in Alertmanager.

From CLI

kubectl get pods -l app=hello
kubectl describe pod <pod-name>
kubectl logs <pod-name> --tail=50
Check recent changes: 
kubectl rollout history deployment hello

🧮 Step 4 — Investigate Locally (Linux & Docker)

  1. Pull broken image locally: 
    docker pull ghcr.io/<your-org>/hello:broken
  2. Run it: 
    docker run --rm -p 8080:8080 ghcr.io/<your-org>/hello:broken
  3. Observe logs and identify issue (e.g., missing dependency, syntax error).
  4. Fix code locally and rebuild: 
    docker build -t ghcr.io/<your-org>/hello:1.1 .
docker push ghcr.io/<your-org>/hello:1.1

🛠 Step 5 — Mitigation in Kubernetes

Rollback quickly to restore service: 

kubectl rollout undo deployment hello
Confirm: 
kubectl get pods -l app=hello
curl http://<hello-service-ip>:8080

🏁 Step 6 — Deploy Fixed Version

Deploy new working image: 

kubectl set image deployment/hello hello=ghcr.io/<your-org>/hello:1.1
kubectl rollout status deployment hello

📊 Step 7 — Verify Recovery

  • Check Grafana dashboard — error rate returns to 0
  • Alert in Alertmanager transitions to Resolved
  • kubectl get pods shows all pods Running

🧾 Step 8 — Postmortem

Example Template: | Section | Details | |----------------|---------| | Incident | SEV-1 — Hello app outage | | Impact | All requests failing with HTTP 500 for 7 min | | Start | 14:03 UTC | | End | 14:10 UTC | | Root Cause | Bad image pushed without health checks | | Mitigation | Rolled back to last known good version | | Fix | Rebuilt image with missing dependency | | Action Items | Add CI build tests, enable canary deployments |

🧪 Bonus Drills

Drill 1 — Dependency Failure

  • Deploy a fake api service that hello depends on.
  • Kill the api deployment.
  • Triage and resolve by redeploying api.

Drill 2 — Resource Starvation

  • Set very low CPU/memory limits.
  • Watch pods get OOMKilled.
  • Fix by increasing resource requests/limits.

Drill 3 — Network Isolation

  • Apply restrictive NetworkPolicy to block DB access.
  • Observe failures.
  • Roll back NetworkPolicy.

✅ Success Criteria

  • [ ] Detected the failure via monitoring
  • [ ] Investigated with both CLI & dashboards
  • [ ] Mitigated quickly via rollback
  • [ ] Deployed fixed version without downtime
  • [ ] Verified recovery in Grafana/Alertmanager
  • [ ] Completed postmortem with actionable follow-ups

A full SRE shift means juggling detection, response, and long-term fixes — this drill gives you that muscle memory.