0/3 nodes are available: how to fix a Pending pod

A pod that stays Pending has not failed. It has not been placed. Nothing has been pulled, nothing has started, and no container has produced a log line to read — which is why every instinct you have for debugging a running workload is useless here.

The entire answer is in one event, and almost everyone reads it wrong.

The fastest check

kubectl describe pod <pod> | tail -20

The last thing printed is the scheduler’s FailedScheduling event. If you want it across the whole cluster at once:

kubectl get events -A --field-selector reason=FailedScheduling --sort-by=.lastTimestamp

That field selector is supported natively by the API server, so it works on a busy cluster where grep over all events would time out.

Where the message comes from

Zoom

The message is a tally, not a reason

This is the part that trips people up:

0/6 nodes are available: 2 Insufficient cpu, 1 node(s) had untolerated taint
{node-role.kubernetes.io/control-plane: }, 3 node(s) didn't match Pod's node
affinity/selector. preemption: 0/6 nodes are available: 6 Preemption is not
helpful for scheduling.

There is no single reason your pod is Pending. The scheduler filtered every node independently and is reporting a histogram of rejection reasons, with a count in front of each one. Six nodes rejected the pod for three different reasons.

So “Insufficient cpu” does not mean the cluster is out of CPU. It means two of six nodes were out; the other four were eliminated by rules that have nothing to do with capacity. Fixing the CPU on those two nodes still leaves you with four nodes you cannot land on.

Read the counts first. The clause with the largest count is the one worth fixing, because it is blocking the most capacity.

Zoom

What each clause means

Insufficient cpu / Insufficient memory

The scheduler compares your pod’s requests — never its limits — against the node’s remaining allocatable. Two things follow from that, and both surprise people.

Limits are invisible to the scheduler. A pod with requests.cpu: 4 and no limit is harder to schedule than one with limits.cpu: 16 and requests.cpu: 100m. If your pods will not schedule on a cluster that looks idle, your requests are the thing to look at, not your actual usage.

Allocatable is smaller than capacity. The kubelet subtracts kube-reserved, system-reserved and the hard eviction threshold before offering anything to the scheduler. On one of our nodes:

kubectl get node <node> -o jsonpath='{.status.capacity.cpu}{" / "}{.status.allocatable.cpu}{"\n"}'

4 / 3830m

Four cores of hardware, 3.83 offered to workloads. On memory the gap is usually far wider — often a gigabyte or more. Sizing requests against the instance type rather than against allocatable is a reliable way to build a cluster where the last pod never fits.

To see what is already committed:

kubectl describe node <node> | grep -A8 "Allocated resources"

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource   Requests      Limits
  --------   --------      ------
  cpu        2527m (65%)    32700m (853%)
  memory     11576Mi (39%)  66154Mi (224%)

The percentages are against allocatable, and the Requests column is the only one the scheduler cares about. Limits at 853% is not a bug — it is what overcommit looks like, and it is exactly why kubectl top will disagree with this table.

node(s) had untolerated taint {key: value}

The node is marked as reserved and your pod has no matching toleration. The taint key is printed in the message, which is usually enough to identify it:

kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'

NAME       TAINTS
worker-1   <none>
worker-2   gpu

Some taints are yours and want a toleration. Others are the cluster telling you something is wrong, and tolerating them is the wrong fix:

If you see not-ready or unreachable in a FailedScheduling message, the pod is not your problem. The node is.

node(s) didn't match Pod's node affinity/selector

A nodeSelector or requiredDuringSchedulingIgnoredDuringExecution affinity matches nothing. In practice this is nearly always a label that does not exist — a typo, a renamed node pool, or a well-known label the cloud provider spells differently from the tutorial you copied.

kubectl get nodes --show-labels
kubectl get nodes -l <your-selector>

The second command is the direct test. If it prints nothing, the scheduler sees nothing either.

pod has unbound immediate PersistentVolumeClaims

Scheduling is blocked on storage. The PVC’s StorageClass has volumeBindingMode: Immediate, so the volume must exist before the pod can be placed, and it does not.

kubectl get pvc <pvc> -o wide
kubectl describe pvc <pvc> | tail -10

If the class is WaitForFirstConsumer instead, this message means something different and more interesting: the PVC is deliberately waiting for the scheduler, and the scheduler is waiting for something else in the same message. Fix the other clause and both resolve.

node(s) exceed max volume count

A per-node attachment limit, not a capacity problem. Cloud providers cap how many disks can attach to one instance, and the limit is often smaller than you expect on small instance types.

Too many pods

The node hit its maxPods limit — 110 by default, and considerably lower on some managed platforms where pod count is bounded by allocatable IP addresses per instance rather than by the kubelet setting.

kubectl get node <node> -o jsonpath='{.status.allocatable.pods}{"\n"}'

node(s) didn't satisfy existing pods anti-affinity rules

Your pod, or a pod already running, has a podAntiAffinity that forbids the combination. The classic version of this is a StatefulSet with anti-affinity on kubernetes.io/hostname and more replicas than nodes: the last replica is permanently Pending by design, and no amount of capacity fixes it.

didn't match pod topology spread constraints

A topologySpreadConstraints block with whenUnsatisfiable: DoNotSchedule is being enforced. Setting it to ScheduleAnyway turns the constraint into a preference, which is usually what people meant when they wrote it.

The preemption half of the message

Everything after preemption: is a second, separate report: having failed to place your pod, the scheduler then checked whether evicting something lower priority would help.

• No preemption victims found for incoming pod — there is nothing running with lower priority than yours to evict.
• Preemption is not helpful for scheduling — evicting things would not fix the reason you were rejected anyway. Taints, affinity and volume limits are not capacity problems, so removing pods changes nothing.

Neither line is an additional failure, and neither is worth debugging. If the first half of the message is solved, the second half disappears with it.

When the message is no nodes available to schedule pods

No count, no clauses. This means the scheduler found zero schedulable nodes at all — every node is cordoned, NotReady, or the cluster genuinely has none.

kubectl get nodes

SchedulingDisabled next to a node means someone ran kubectl cordon, or a drain is in progress and was never completed.

A Pending pod that has already been placed

Not every Pending pod is a scheduling failure. Once spec.nodeName is set, the pod is scheduled and Pending now means the kubelet is still working — pulling an image, mounting a volume, or waiting on an init container.

kubectl get pod <pod> -o jsonpath='{.spec.nodeName}{"\n"}'

If that prints a node name, stop looking at the scheduler. The events on the pod will name the real stage, and if the image is the problem you are heading for ImagePullBackOff and ErrImagePull. The Pending phase and the Pending shown in kubectl’s STATUS column are also not quite the same thing — we pulled that apart in how kubectl decides what to put in the STATUS column.

Stopping it happening again

• Set requests from observed usage, and set them deliberately. Most Insufficient cpu incidents are one service requesting an order of magnitude more than it uses, copied from a manifest where it was already wrong.
• Never tolerate a pressure or not-ready taint. Those are the cluster refusing work it cannot do. Tolerating them converts a Pending pod into an evicted one.
• Alert on Pending duration, not on Pending. Pods are briefly Pending on every deploy. A pod Pending for five minutes is an incident and nothing in a default install will tell you.
• Check anti-affinity against your real node count before scaling a StatefulSet. Replicas greater than nodes plus hostname anti-affinity is a permanent Pending you built yourself.