Runbooks
One symptom per page. Confirm it, follow the branch the output points to, and stop when the check says the incident is over.
Alert fixed, users still see errors: a second fault the alert did not cover (Kubernetes)
The alerted pod is explained, yet users still get errors: a second, unalerted fault. Scope past the alert with real lab output from two faults at once.
Sidecar OOMKilled, app healthy: the pod drops out of its Service (Kubernetes)
A sidecar OOMKilled under its own 16Mi limit left a healthy api 1/2 Ready and its Service empty. Find the container, fix its limit, verify. Lab-tested.
Name does not resolve with dnsPolicy None: the search domains are wrong (Kubernetes)
New pods log Name does not resolve for a Service that exists. The template set dnsPolicy None with search domains for another cluster domain. Lab-tested.
One in three requests to a Service fails: its selector also matches pods from another Deployment (Kubernetes)
A Service selects pods by label, not by Deployment. Its endpoints list one pod more than the Deployment runs and that pod returns 500. Confirm and fix it.
OOMKilled on a steady interval: a memory leak, not a small limit (Kubernetes)
OOMKilled on a steady interval with clean logs and no spec change is a leak. Tell it from a small limit by cycle length and growth, then fix the code.
Service has endpoints but connections are refused: targetPort does not match the container port (Kubernetes)
Endpoints list the pod IPs, yet nginx logs Connection refused and returns 502: the Service targetPort points where nothing listens. Confirm it and fix it.
ConfigMap change not applied to all pods: replicas serve different configs (Kubernetes)
After a ConfigMap edit, only pods started since the edit run the new config; others keep what they loaded at startup. Compare per-pod logs, fix, restart.
Pod stuck in Init:0/1: an init container never finishes (Kubernetes)
A pod stuck in Init:0/1 has an init container that never exits, often a wait loop on a missing Service. Read its logs, check the Service, fix the spec.
Pod Pending with FailedScheduling: Insufficient cpu, the request fits no node (Kubernetes)
A pod Pending with FailedScheduling Insufficient cpu asks for more CPU than any node has. Read the event, compare request and nodes, fix the request.
PVC Pending: the StorageClass does not exist and the dependent app fails (Kubernetes)
A PVC stays Pending when its storageClassName names a class the cluster lacks. The pod never schedules and whatever depends on it fails. Lab-tested.
Connections time out: a NetworkPolicy blocks the app from its dependency (Kubernetes)
Requests time out while every pod is Running: a NetworkPolicy lets only other pods reach the dependency. Read the error, find the policy, fix it.
Exceeded quota: a ResourceQuota blocks the rollout's surge pod (Kubernetes)
A Deployment rollout that never progresses, with FailedCreate and exceeded quota on the new ReplicaSet. Confirm the quota, give it headroom, verify.
CreateContainerConfigError: the Secret or ConfigMap is missing (Kubernetes)
CreateContainerConfigError means a Secret or ConfigMap the pod references is missing. Read the event, create the object or roll back, verify. Lab-tested.
nginx [emerg] unknown directive: a ConfigMap typo crash-loops the pod (Kubernetes)
nginx [emerg] unknown directive means a config line nginx does not know, usually a typo in a mounted ConfigMap. Find the line, fix it, restart, check.
Liveness probe failed: wrong port, healthy app restarted (Kubernetes)
Liveness probe failed with connection refused while the app serves traffic: the probe points at a port nothing listens on. Confirm, fix the port, verify.
Name does not resolve: a wrong Service hostname crash-loops the app (Kubernetes)
The app exits with Name does not resolve because an env var names a Service that does not exist. Confirm it, rule out CoreDNS, fix the name. Lab-tested.
Service has no endpoints: the selector matches no pods, upstream returns 502 (Kubernetes)
A Service with no endpoints has a selector that matches no pods, so its ClusterIP refuses connections and the proxy returns 502. Confirm it and fix it.
OOMKilled (exit code 137): container exceeded its memory limit (Kubernetes)
OOMKilled means the kernel killed a container for exceeding its memory limit. Confirm it, find the change behind it, fix and verify, with real lab output.
ImagePullBackOff: the image could not be pulled and Kubernetes is backing off (Kubernetes)
ImagePullBackOff means the kubelet could not pull the image and is retrying, up to 5 minutes apart. Read the event, fix the reference, verify. Lab-tested.