Kubernetes Troubleshooting Runbook

Overview

This runbook covers the most common Kubernetes production issues and their resolution paths. Designed to be followed at 3am when something is on fire.

Quick Reference: First 60 Seconds

# Cluster health
kubectl get nodes
kubectl top nodes

# Namespace overview
kubectl get pods -n <namespace> --sort-by='.status.startTime'
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20

# Problem pod
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --tail=100
kubectl logs <pod> -n <namespace> --previous  # previous crashed container

---

Issue: CrashLoopBackOff

Symptoms: Pod keeps restarting, back-off delay increasing.

Diagnosis:

kubectl describe pod <pod> -n <ns>
# Look at: Exit Code, Reason, Last State

kubectl logs <pod> -n <ns> --previous
# Shows logs from the crashed container

Common Causes & Fixes:

Exit Code	Meaning	Fix
1	Application error	Check app logs, fix the bug
137	OOM Killed	Increase resources.limits.memory
143	SIGTERM (graceful)	Check readiness probe, increase terminationGracePeriodSeconds
126	Permission denied	Check container user, file permissions
127	Command not found	Check image, entrypoint, PATH

---

Issue: Pod Stuck in Pending

Diagnosis:

kubectl describe pod <pod> -n <ns>
# Look at Events section for scheduling failures

Common Causes:

Event Message	Cause	Fix
Insufficient cpu/memory	Cluster out of resources	Scale nodes or reduce requests
no nodes available to schedule	Node selector/affinity mismatch	Check nodeSelector, tolerations
persistentvolumeclaim not found	PVC not bound	Check StorageClass, PVC status
0/N nodes are available: taint	Taints blocking scheduling	Add tolerations or remove taint

# Check node resources
kubectl describe nodes | grep -A5 "Allocated resources"

# Check PVC
kubectl get pvc -n <ns>

---

Issue: OOMKilled (Exit Code 137)

Diagnosis:

kubectl describe pod <pod> -n <ns> | grep -A3 "Last State"
# Reason: OOMKilled

kubectl top pod <pod> -n <ns>

Fix:

1. Increase memory limits:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"  # Increase this

2. If the app genuinely needs more memory, profile it (heap dump, pprof).
3. If it's a memory leak, fix the application code.

Prevention: Set up alerts on container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.8.

---

Issue: Service Not Reachable

Diagnosis flow:

# 1. Is the pod running?
kubectl get pods -l app=<app> -n <ns>

# 2. Is the service pointing to the pod?
kubectl get endpoints <service> -n <ns>
# If empty → labels don't match

# 3. Can you reach from inside the cluster?
kubectl run tmp --rm -it --image=busybox -- wget -qO- http://<service>.<ns>.svc:port

# 4. Is Ingress configured?
kubectl describe ingress <ingress> -n <ns>

Common Issues:

Problem	Check
No endpoints	Service selector doesn't match pod labels
Connection refused	Pod is up but app not listening on expected port
502/504 from Ingress	Pod failing health checks → Ingress removes it
DNS not resolving	CoreDNS pods healthy? kubectl get pods -n kube-system -l k8s-app=kube-dns

---

Issue: ImagePullBackOff

kubectl describe pod <pod> -n <ns>
# Look for: Failed to pull image

Fixes:

Cause	Fix
Image doesn't exist	Check tag, repo URL
Private registry, no auth	Create imagePullSecret, reference in pod spec
Rate limited (Docker Hub)	Use authenticated pulls or mirror to private registry
Wrong platform (arm64 vs amd64)	Build multi-arch or specify correct image

---

Issue: High Pod Restart Count

# Find pods with high restart counts
kubectl get pods -n <ns> --sort-by='.status.containerStatuses[0].restartCount'

# Check why
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous

Common pattern: Liveness probe failing → kubelet kills the pod → restarts.

Fix: Tune liveness probe (increase initialDelaySeconds, timeoutSeconds, failureThreshold). Make sure the probe endpoint is lightweight.

---

Issue: Node NotReady

kubectl get nodes
kubectl describe node <node>

# Check kubelet
# (SSH to node)
systemctl status kubelet
journalctl -u kubelet --since "10 minutes ago"

Common Causes:

• Kubelet crashed or resource exhaustion
• Network partition (node can't reach API server)
• Disk pressure (kubectl describe node → Conditions)
• Certificate expiry

---

Issue: Deployment Not Rolling Out

kubectl rollout status deployment/<name> -n <ns>
kubectl describe deployment <name> -n <ns>
kubectl get replicasets -n <ns> -l app=<app>

Common causes:

• New pods failing health checks → old pods stay, new pods don't become Ready
• maxUnavailable: 0 + no extra capacity → deadlock
• Resource quota exceeded in namespace

---

Useful Aliases

alias k=kubectl
alias kgp='kubectl get pods'
alias kgn='kubectl get nodes'
alias kdp='kubectl describe pod'
alias kl='kubectl logs'
alias klp='kubectl logs --previous'
alias ktp='kubectl top pods'
alias ktn='kubectl top nodes'

Monitoring Queries (Prometheus)

# Pods in CrashLoopBackOff
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} > 0

# OOM kills
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} > 0

# Pending pods
kube_pod_status_phase{phase="Pending"} > 0

# High restart count
kube_pod_container_status_restarts_total > 5

# Node not ready
kube_node_status_condition{condition="Ready",status="true"} == 0