KubeGlanceDownload
Pod is Running but 0/1 READY: what the READY column is actually counting

troubleshooting

Pod is Running but 0/1 READY: what the READY column is actually counting

Running but 0/1 READY, no restarts: READY and STATUS are separate computations. What READY counts, why it lies, and why the Service skips the pod.

· 9 min read

Running and 0/1 in the same row are two different questions answered by two different pieces of code. STATUS says the containers are up. READY says how many of them are passing readiness right now. A readiness failure never changes the phase and never restarts anything, so a pod can sit at 0/1 Running with RESTARTS 0 for as long as you let it — and for all that time, no Service sends it traffic.

NAME      READY   STATUS    RESTARTS   AGE
web-503   0/1     Running   0          7m8s
web-ok    1/1     Running   0          7m8s

Everything below was reproduced on Kubernetes 1.36.1.

The fastest check

Ask the pod’s conditions, not the table:

kubectl get pod <pod> -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
PodReadyToStartContainers	True		
Initialized	True		
Ready	False	ContainersNotReady	containers with unready status: [app]
ContainersReady	False	ContainersNotReady	containers with unready status: [app]
PodScheduled	True		

The Ready row is the one Services act on, and its reason sorts you into one of two branches:

  • ContainersNotReady — a named container is failing readiness, or has not passed its startup probe yet. The message names it. Go to the events.
  • ReadinessGatesNotReady — every container is fine, and something outside the kubelet has not signed off. This is the case where READY can say 1/1 and the pod still gets no traffic.

What READY counts, and what it does not

The table kubectl get pods prints is built on the API server by printPod in pkg/printers/internalversion/printers.go (release-1.36). The READY cell is fmt.Sprintf("%d/%d", readyContainers, totalContainers) at line 1072, and the two numbers come from different places:

  • The denominator comes from the spec. len(pod.Spec.Containers) (line 927), plus one for every init container with restartPolicy: Always — a native sidecar (lines 956-962).
  • The numerator comes from the status. It walks status.containerStatuses and counts each one that is both ready and in state running (line 1043), plus native sidecars that have started: true and ready: true (lines 986-990).

STATUS comes out of the same function from a separate path: the pod phase, overridden by any container’s waiting or terminated reason. Nothing in that path looks at readiness. A container that is running and failing its readiness probe contributes Running to STATUS and 0 to READY, and both are true.

Two consequences follow straight from the code, and both catch people:

  1. READY is not the pod’s Ready condition. It never reads status.conditions. Readiness gates are shown in a separate column, and only with -o wide.
  2. Numerator and denominator can disagree about which containers exist. One counts status entries, the other counts spec entries. That is how 3/2 happens, below.

Why the pod never becomes Ready

Readiness is decided by the kubelet, published as conditions, and consumed by the EndpointSlice controller. kubectl only reads the result.

The kubelet writes three layers of readiness. Services act on the last one; the READY column shows the first.

The layering matters. ContainersReady is every container and native sidecar ready. Ready is ContainersReady plus every readiness gate True (pkg/kubelet/status/generate.go, release-1.36). The EndpointSlice controller copies the pod’s Ready condition into the endpoint’s ready field (staging/src/k8s.io/endpointslice/utils.go, line 39 and 43), unless the Service sets publishNotReadyAddresses.

There is no timeout anywhere in that chain. A liveness probe that fails restarts the container; a readiness probe that fails is simply reported, forever.

The four ways to get there

A failing readiness probe

The common case. The kubelet records every failure as an Unhealthy event, and the message tail tells you which of three things happened:

kubectl describe pod web-503
  Warning  Unhealthy  2m4s (x65 over 7m8s)  kubelet            spec.containers{app}: Readiness probe failed: HTTP probe failed with statuscode: 503

The app answered and said no. The probe is working; the application’s own health check is failing, usually on a dependency — a database, a cache, a config file. This is an application question, not a Kubernetes one.

Readiness probe failed: Get "http://10.244.0.62:8080/cgi-bin/slow": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The app accepted the connection and did not respond within timeoutSeconds (default 1). This is the kubelet timing out against your pod, not kubectl timing out against the API server — same Go wording, unrelated failure. If you see context deadline exceeded from a kubectl command, that is a connection or TLS problem instead. Here, either the endpoint is genuinely slow (a health check that queries the database is the usual culprit) or timeoutSeconds is too tight.

Readiness probe failed: Get "http://10.244.0.63:9090/": dial tcp 10.244.0.63:9090: connect: connection refused

Nothing is listening on that port. Either the probe port is wrong or the process has not bound yet. If it persists, compare the probe’s port against what the container actually listens on.

Note the RESTARTS 0 on all three. If the counter is climbing too, you have a liveness failure or a crash, and CrashLoopBackOff is the right article.

A startup probe that has not passed

While a startup probe is failing, the kubelet does not run the readiness probe at all (pkg/kubelet/prober/worker.go lines 340-344, release-1.36: “Disable other probes until container has started”) and the container is held at ready: false. The pod looks exactly like the case above, and the difference is the started field:

kubectl get pod slow-start -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}started={.started}{"\t"}ready={.ready}{"\n"}{end}'
app	started=false	ready=false

The events say Startup probe failed: — for an exec probe whose command prints nothing, with nothing after the colon:

  Warning  Unhealthy  3s (x18 over 2m53s)  kubelet            spec.containers{app}: Startup probe failed:

A startup probe with a generous failureThreshold × periodSeconds is designed to hold readiness for a long time. That is the point of it, and it is also why a slow-starting pod at 0/1 for ten minutes may be behaving exactly as configured. Once it passes, readiness takes over and READY flips without a restart.

A readiness gate nobody satisfies

spec.readinessGates adds a condition type that must be True before the pod is Ready. The kubelet never writes that condition; an external controller does — typically a cloud load-balancer controller confirming the pod is registered as a target. If that controller is missing, misconfigured or failing, this is what you get:

kubectl get pod gated -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP            NODE                        NOMINATED NODE   READINESS GATES
gated   1/1     Running   0          30s   10.244.0.70   glance-blog-control-plane   <none>           0/1

1/1 READY, and not ready. The READY column counts containers; the gate is in the last column, which the default output does not print. The conditions say so plainly:

Ready	False	ReadinessGatesNotReady	corresponding condition of pod readiness gate "example.com/lb-registered" does not exist.
ContainersReady	True		

does not exist means no controller has ever written the condition. If the message is instead the status of pod readiness gate ... is not "True", but False, the controller is running and has decided against the pod — read its logs. Writing the condition by hand proves the mechanism, and is a test, not a fix:

kubectl patch pod gated --subresource=status -p '{"status":{"conditions":[{"type":"example.com/lb-registered","status":"True"}]}}'
NAME    READY   STATUS    RESTARTS   AGE   IP            NODE                        NOMINATED NODE   READINESS GATES
gated   1/1     Running   0          37s   10.244.0.70   glance-blog-control-plane   <none>           1/1

The kubelet recomputes Ready on its next status sync and the pod joins the Service.

A native sidecar that is not ready

A native sidecar is an init container with restartPolicy: Always. It adds one to the denominator, and its readiness gates the whole pod:

NAME           READY   STATUS    RESTARTS   AGE
with-sidecar   1/2     Running   0          3m3s

The pod has one entry in spec.containers. The 2 is the sidecar, and so is the missing 1:

kubectl get pod with-sidecar -o jsonpath='{range .status.initContainerStatuses[*]}{.name} started={.started} ready={.ready}{"\n"}{end}{range .status.containerStatuses[*]}{.name} started={.started} ready={.ready}{"\n"}{end}'
proxy started=true ready=false
app started=true ready=true

The conditions name it: containers with unready status: [proxy]. If you are used to reading 1/2 as “the app is down”, look at the init containers first — a service-mesh proxy or log shipper running as a native sidecar is now part of the pod’s readiness. Ordinary init containers (without restartPolicy: Always) are not counted in READY at all.

The Service already excludes it, and get endpointslices hides that

Not ready means no traffic. The obvious way to check is misleading:

kubectl get endpointslices -l kubernetes.io/service-name=web
NAME        ADDRESSTYPE   PORTS   ENDPOINTS                 AGE
web-gft4g   IPv4          8080    10.244.0.56,10.244.0.57,10.244.0.70   7m8s

All three addresses are listed. 10.244.0.57 is the 0/1 pod, and 10.244.0.70 is the gated one. The ENDPOINTS column is built by formatDiscoveryEndpoints (printers.go line 848, release-1.36), which prints every address in the slice and never reads its conditions. An EndpointSlice keeps not-ready pods as entries with ready: false; kube-proxy is what skips them. To see what actually receives traffic:

kubectl get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.targetRef.name}{"\t"}{.addresses[0]}{"\t"}{.conditions.ready}{"\n"}{end}'
web-ok	10.244.0.56	true
web-503	10.244.0.57	false
gated	10.244.0.70	false

gated is there too, at ready: false despite showing 1/1 in kubectl get pods. The deprecated Endpoints object gets this right by accident, because it splits not-ready addresses into a separate field the printer ignores:

kubectl get endpoints web
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME   ENDPOINTS          AGE
web    10.244.0.56:8080   7m8s

If every pod behind a Service is in this state, the Service has no ready endpoints and clients get connection refused or a 503 from whatever sits in front — with every pod showing Running.

The 3/2 case

The numerator walks status.containerStatuses; the denominator counts spec.containers. If a status reports a container the spec does not declare, READY exceeds its own total. A standard kubelet reports the containers the spec declares, so you will not see this on an ordinary node, but other node implementations can do it: k9s#4145 reports it on Tencent Cloud TKE virtual nodes, where a provider-managed logging container appears in containerStatuses with no matching entry in the spec.

We reproduced it by writing such a status directly to a two-container pod bound to a node with no kubelet, so nothing overwrote it:

NAME           READY   STATUS    RESTARTS   AGE
extra-status   3/2     Running   0          0s

(The pod garbage collector deletes a pod bound to a node that does not exist, so this lasted under a minute. The printer does not care how the status got there.)

It is not a kubectl bug so much as the two halves of the fraction being honest about different things. If you see it, the extra container is real on the provider’s side, and status.containerStatuses[*].name will tell you what it is. Tools that colour a row red whenever the numbers differ will flag a healthy pod.

Fixing it

  • Readiness probe failing: fix what the probe is checking, or fix the probe. Test the endpoint from inside the pod with the same path and port the probe uses. If the endpoint depends on a database, decide whether you really want every replica to drop out of the Service when the database blips — that turns a partial outage into a total one.
  • Timeouts: raise timeoutSeconds or make the health endpoint cheap. A readiness endpoint should answer from memory.
  • Startup probe: wait, or shorten it if the budget is far longer than the app needs. started=false with nothing wrong in the logs is a configuration choice, not a fault.
  • Readiness gate: find the controller that owns the condition type (the prefix usually names it) and check it is running and has permission to patch pod status.
  • Native sidecar: treat its readiness probe as part of the application’s. A proxy that never becomes ready takes the whole pod out of the Service.

Preventing it

  • Alert on the Ready condition, not on STATUS. A dashboard that counts Running pods will show full health through a total readiness outage.
  • Read -o wide when READY and behaviour disagree. The READINESS GATES column is the only place a gate shows up in the table.
  • Keep readiness and liveness checks different. A readiness check may depend on downstream services; a liveness check should not, or a database outage restarts every pod you have.
  • Check .conditions.ready, never the ENDPOINTS column, when proving a pod is or is not receiving traffic.

For how STATUS itself is derived — the other half of that table row — see how kubectl computes a pod’s STATUS column.

READY, readiness gates and the conditions behind them sit on the same screen. KubeGlance is a native Kubernetes client for iPhone and iPad, with a full Mac app on the same core — pod conditions, probe events and container states without a jsonpath expression.

Get KubeGlance
#readiness #probes #pods #kubectl #endpointslices #troubleshooting

The Kubernetes dashboard that fits in your pocket

KubeGlance is a native Kubernetes client for iPhone and iPad — the real dashboard, not a companion — with a full Mac app on the same core. Pods, workloads, logs and events, straight from your kubeconfig.

Download KubeGlance