Skip to content

Lab — Incident Response Simulation

Difficulty: Intermediate · Estimated time: 45-60 min

Simulate a real-world production incident and practice the full response process: detect → triage → mitigate → resolve → review.

🎯 Objective

Work through a simulated outage to: - Identify the issue from monitoring and logs - Communicate effectively with the team - Apply a quick mitigation - Perform a safe resolution - Write a postmortem

🛠 Lab Setup

Prerequisites

  • Kubernetes cluster (minikube, kind, or cloud)
  • kubectl CLI access
  • Sample app deployed from Lab 6 — Deploy an App

Simulated Service

We’ll use the hello app from previous labs and introduce a failure.

🧨 Step 1 — Deploy the Healthy App

Deploy version 1.0: 

kubectl apply -f deployment.yaml
kubectl rollout status deployment hello
kubectl get pods -l app=hello

🚨 Step 2 — Simulate an Incident

Edit deployment.yaml to use a broken image: 

image: ghcr.io/<your-org>/hello:broken
Apply the change: 
kubectl apply -f deployment.yaml

Expected outcome: - Pods crashloop - Probes fail - Error rate spikes

🔍 Step 3 — Detection

Metrics: - Use kubectl top pods (CPU/mem anomalies) - Check Grafana dashboards (if integrated) - Run PromQL for error rate: 

sum(rate(http_requests_total{status!~"2.."}[5m]))

Logs: 

kubectl logs -l app=hello --tail=50

🧮 Step 4 — Triage

Answer: 1. Scope — Which namespaces/services are impacted? 

kubectl get pods --all-namespaces | grep -i crash
2. Recent changes — Deployments in last 30m: 
kubectl get events --sort-by=.lastTimestamp | tail
3. Dependencies — Is this issue isolated to one service or cascading?

🛠 Step 5 — Mitigation

Rollback to last working version: 

kubectl rollout undo deployment hello
kubectl rollout status deployment hello

Scale up temporarily to handle backlog: 

kubectl scale deployment hello --replicas=5

🏁 Step 6 — Resolution

Confirm: 

kubectl get pods -l app=hello
kubectl logs -l app=hello | tail
curl -I http://<service-ip>:8080
- All pods healthy - Probes passing - Error rate back to normal

🧾 Step 7 — Postmortem

Use the Postmortem Template and record: - Incident title, severity - Impact metrics - Root cause - Timeline - Actions to prevent recurrence

🧪 Bonus Challenge

1. Break a Dependency

  • Deploy a second service (api) that hello depends on.
  • Stop the api deployment and observe the impact on hello.

2. Network Failure

  • Apply a restrictive NetworkPolicy that blocks hello from talking to api.
  • Diagnose using kubectl describe networkpolicy.

3. Pod Resource Starvation

  • Set very low CPU/memory limits in the deployment.
  • Watch pods get OOMKilled and fix by adjusting resources.

✅ Success Criteria

  • [ ] Detected the issue via metrics/logs
  • [ ] Determined scope & severity
  • [ ] Applied a safe rollback or mitigation
  • [ ] Verified full recovery
  • [ ] Completed a postmortem with actionable follow-ups

Incidents are a matter of “when,” not “if.” Practicing the process is what makes you reliable under pressure.