Skip to content

Lab — Kubernetes Readiness & Liveness Probes

Difficulty: Beginner · Estimated time: 30-45 min

Configure readiness and liveness probes on a Deployment, then deliberately break them to see exactly what Kubernetes does in each case — the two failure modes look similar in the dashboard but mean very different things to an on-call engineer.

🎯 Objective

  • Deploy an app with a /healthz endpoint you can toggle on and off
  • Add a readinessProbe and watch a pod get pulled out of Service traffic without being restarted
  • Add a livenessProbe and watch kubelet restart the container instead
  • Understand why conflating the two is a common cause of cascading outages

🛠 Prerequisites

  • Kubernetes cluster with kubectl access
  • Any image that serves a toggleable health endpoint (nginx + a mounted static file works fine, or reuse the hello app from the earlier lab)

🧭 How the two probes differ

flowchart TD
    A[Pod starts] --> B{Readiness probe}
    B -- passing --> C[Added to Service endpoints<br/>receives traffic]
    B -- failing --> D[Removed from Service endpoints<br/>no traffic, pod keeps running]
    C --> E{Liveness probe}
    D --> E
    E -- passing --> C
    E -- failing --> F[kubelet kills & restarts container]
    F --> A
  • Readiness answers "can this pod take traffic right now?" — failing it just removes the pod from the Service's endpoint list. No restart.
  • Liveness answers "is this process still healthy, or is it stuck?" — failing it gets the container killed and restarted. Get this wrong (e.g. tying liveness to a downstream dependency) and one flaky dependency can restart-loop your entire fleet.

📦 Step 1 — Deploy without probes

kubectl create deployment probes-demo --image=nginxdemos/hello --port=80
kubectl expose deployment probes-demo --port=80 --target-port=80
kubectl get endpoints probes-demo

Note the pod IP is already in the endpoint list — with no readiness probe, Kubernetes assumes "running container" means "ready for traffic" the moment it starts.

✅ Step 2 — Add a readiness probe

kubectl patch deployment probes-demo --type=json -p='[
  {"op": "add", "path": "/spec/template/spec/containers/0/readinessProbe", "value": {
    "httpGet": {"path": "/", "port": 80},
    "initialDelaySeconds": 3,
    "periodSeconds": 5
  }}
]'
kubectl rollout status deployment/probes-demo
kubectl get pods -l app=probes-demo -o wide

The READY column now shows 1/1 only once the probe has passed at least once.

💥 Step 3 — Break readiness and watch traffic drain

# Point the probe at a path that doesn't exist
kubectl patch deployment probes-demo --type=json -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/does-not-exist"}
]'

kubectl get pods -l app=probes-demo -w

Watch READY drop to 0/1 after a few failed probes. Now check the Service:

kubectl get endpoints probes-demo

The endpoint list is empty — the pod is still running (kubectl get pods shows Running, not CrashLoopBackOff), it's just no longer receiving traffic. This is the behavior you want during a slow startup or a temporary dependency outage: fail closed, don't restart.

♻️ Step 4 — Add and break a liveness probe

kubectl patch deployment probes-demo --type=json -p='[
  {"op": "add", "path": "/spec/template/spec/containers/0/livenessProbe", "value": {
    "httpGet": {"path": "/does-not-exist", "port": 80},
    "initialDelaySeconds": 3,
    "periodSeconds": 5,
    "failureThreshold": 3
  }}
]'

kubectl get pods -l app=probes-demo -w

This time watch RESTARTS climb instead of READY staying pinned at 0/1 — kubelet is killing and restarting the container each time the liveness probe fails its threshold. Run kubectl describe pod -l app=probes-demo and look at the Events section to see the Liveness probe failed / Killing container entries.

🧹 Step 5 — Fix it and clean up

kubectl patch deployment probes-demo --type=json -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/path", "value": "/"}
]'
kubectl rollout status deployment/probes-demo

# Cleanup
kubectl delete deployment probes-demo
kubectl delete service probes-demo

✅ Success Criteria

  • [ ] Deployed a pod with no probes and observed it receives traffic immediately on start
  • [ ] Added a readiness probe and confirmed READY gates on it
  • [ ] Broke readiness and confirmed the pod stays Running but drops out of Service endpoints
  • [ ] Broke liveness and confirmed the container gets restarted instead
  • [ ] Can explain, in one sentence each, when you'd want readiness-only vs. liveness failing

A readiness probe protects your users from a pod. A liveness probe protects a pod from itself. Wiring a downstream dependency into liveness turns one bad dependency into a restart storm — keep liveness checks local to the process.