Incident Management¶

A structured approach to handling production incidents: detect, respond, resolve, and learn.

🎯 Goals¶

Minimize MTTR (Mean Time to Restore)
Reduce user impact
Ensure clear communication
Learn from every incident via postmortems

🔄 Lifecycle¶

Detect
Automated: monitoring & alerting (Prometheus, Datadog, CloudWatch, Splunk)
Manual: customer reports, synthetic checks
Triage
Confirm incident validity
Assign severity
Page on-call responders
Mitigate
Short-term fixes to reduce impact
Examples: rollback, scale up, disable feature flag, route traffic
Resolve
Apply a permanent fix
Verify stability
Review
Conduct a blameless postmortem
Create action items to prevent recurrence

🛑 Severity Levels (Example)¶

Severity	User Impact	Example	Response Time
SEV-1	Major outage, all users impacted	API down globally	Immediate (< 5 min)
SEV-2	Partial outage, significant degradation	EU users see 5xx errors	< 15 min
SEV-3	Minor outage, workaround available	Payments delayed, retried later	< 1h
SEV-4	Low impact, non-urgent	UI glitch, minor metric gap	< 1 business day

📢 Communication¶

Channels - War room (Zoom, Slack, Teams) - Incident channel (#inc-YYYY-MM-DD-shortname) - Status page for customers - Internal stakeholder updates

Roles - Incident Commander (IC) — manages process - Tech Lead — leads technical investigation - Comms Lead — handles stakeholder updates - Scribe — records timeline/events

📊 Detection & Alerting Examples¶

Prometheus

groups:
  - name: api.rules
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) /
              sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "High HTTP 5xx error rate"
          description: "Error rate >5% for 5m. Check service health and logs."

Splunk

index=prod sourcetype=nginx status=500
| stats count by service, host
| sort -count

🛠 Mitigation Examples¶

Rollback a Deployment

kubectl rollout undo deployment/api -n prod
kubectl rollout status deployment/api -n prod

Scale Up to Handle Load

kubectl scale deployment/api -n prod --replicas=8

Disable a Feature Flag

curl -X POST https://flags.prod.example.com/api/v1/disable/feature-x \
  -H "Authorization: Bearer $TOKEN"

🧾 Postmortem Template¶

Title: SEV-1 — API Outage — 2025-08-09
Summary: Short description of what happened, impact, and fix.

Impact: - Start: YYYY-MM-DD HH:MM UTC - End: YYYY-MM-DD HH:MM UTC - Duration: X min - Affected users: % or count - Business impact: (optional)

Root Cause: - What actually broke

Trigger: - What started the incident

Timeline: | Time (UTC) | Event | |------------|-------| | 13:14 | Alert fired | | 13:15 | IC assigned | | 13:17 | Rolled back to v1.3.4 | | 13:42 | Errors cleared |

What Went Well: - Early detection - Clear ownership

What Needs Improvement: - Test coverage gap - Missing runbook for X

Action Items: - [ ] Add canary check for service Y - [ ] Improve alert thresholds - [ ] Document rollback steps

📋 On-Call Checklist¶

When an alert fires: 1. Check if alert is valid 2. Acknowledge alert in paging tool 3. Create/join incident channel 4. Assign roles (IC, Tech Lead, Comms Lead) 5. Triage scope and impact 6. Mitigate → Resolve 7. Log timeline events 8. Schedule postmortem

🔍 Triage Command Snippets¶

Check Pods & Events

kubectl get pods -n prod --sort-by=.status.startTime
kubectl describe pod api-xxx -n prod
kubectl get events -n prod --sort-by=.lastTimestamp | tail

Check Recent Deployments

kubectl rollout history deployment/api -n prod

Quick Service Health

curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
  https://api.prod.example.com/healthz

✅ Readiness Checklist¶

[ ] Defined SEV levels & escalation paths
[ ] On-call rotation documented
[ ] War room procedure tested
[ ] Runbooks for top 5 risky systems
[ ] Status page + comms channels ready
[ ] Postmortem culture: blameless, actionable
[ ] Observability wired for critical paths

Incidents are inevitable — chaos is optional.