Kubernetes Troubleshooting

Kubernetes workload triage

Work from symptom to cause with read-only commands first. Never delete, restart or scale anything until the cause is understood and the user agrees.

1. Establish context

• Confirm the cluster and namespace: kubectl config current-context and the namespace the user means. If the context looks like production, say so before running anything.
• Get the picture in one shot:

kubectl get pods -n <ns> -o wide
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -30

2. Branch on the pod status

Pending

• kubectl describe pod <pod> and read the Events block.
• Insufficient cpu/memory → requests too high or cluster full; check kubectl top nodes and the autoscaler (Karpenter/Cluster Autoscaler) logs.
• didn't match node affinity/selector or untolerated taint → scheduling constraints.
• pod has unbound PersistentVolumeClaims → check the PVC, StorageClass and, on EKS, that the volume's AZ matches a node's AZ.

ImagePullBackOff / ErrImagePull

• Typo in image/tag, tag no longer exists, or registry auth (imagePullSecrets, IRSA/node role for ECR, ECR in another account).

CrashLoopBackOff

• Logs of the previous container: kubectl logs <pod> -c <container> --previous.
• Exit code from kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState}':
  • 137 with reason OOMKilled → memory limit too low or a leak; compare limit vs kubectl top pod and the app's heap settings (JVM -XX:MaxRAMPercentage, Node --max-old-space-size).
  • 137 without OOMKilled → killed by liveness probe or eviction.
  • 1/2 → application error; read the logs, then check config: env vars, mounted ConfigMaps/Secrets, missing files.
• Check whether the liveness probe is killing a slow starter: initialDelaySeconds / startupProbe.

Running but not Ready / Service returning 5xx

• Readiness probe failures in describe.
• kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>: no endpoints means the selector doesn't match pod labels or no pod is Ready.
• Port mismatch: Service targetPort vs container port.
• From inside the cluster: kubectl run tmp --rm -it --image=nicolaka/netshoot -- bash then curl the service and dig its DNS name.
• Ingress/ALB: target group health in AWS, ingress annotations, security groups.

Evicted / node problems

• kubectl describe node <node>: MemoryPressure, DiskPressure, PIDPressure.
• Look for noisy neighbours without limits on the same node.

3. Correlate with change

Most incidents follow a change. Check kubectl rollout history deploy/<name>, the image tag currently running vs the previous ReplicaSet, recent ConfigMap/Secret edits and recent cluster upgrades.

4. Report

Reply with:

1. Cause — one sentence, with the evidence (log line, event, exit code).
2. Fix — the smallest change that resolves it, as a manifest diff or command.
3. Prevent — one item (alert, probe tuning, resource request, policy).

If the evidence is inconclusive, say what is missing and the next command to run rather than guessing.

Rightsizing requests and limits

Gather usage

Prefer at least 7 days of data that includes a peak period.

# p95 CPU (cores) per container over 7d
quantile_over_time(0.95, sum by (namespace, pod, container) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))[7d:5m])

# max working-set memory per container over 7d
max_over_time(container_memory_working_set_bytes{container!=""}[7d])

# CPU throttling ratio
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/ sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total[5m]))

Without Prometheus, use kubectl top pod --containers sampled at peak, and say the recommendation is lower confidence. If VPA runs in recommendation mode, read kubectl describe vpa.

Rules of thumb

• CPU request ≈ p95 usage. It drives scheduling and fair share.
• CPU limit: often better unset for latency-sensitive services, since CFS throttling hurts tail latency. Keep it where the platform requires limits or for batch/noisy workloads. Throttling above ~10% at peak is a signal.
• Memory request ≈ max working set + 15–25% headroom.
• Memory limit = memory request for most services (Guaranteed for memory, predictable OOM behavior). Memory is not compressible: being over the limit means OOMKilled.
• Runtime awareness: JVM heap should be a percentage of the limit (-XX:MaxRAMPercentage=70), Node --max-old-space-size below the limit, Go GOMEMLIMIT around 90% of the limit.
• Keep HPA in mind: HPA on CPU utilization is relative to the request. Changing requests changes when it scales.

Output

A table per container: current request/limit, observed p95/max, recommended request/limit, and the reason. Then the YAML patch. Flag the ones where the change is large (more than 2× either way) so the user rolls them out gradually.