Pod stuck in ContainerCreating: what the kubelet is still waiting for

ContainerCreating is not an error, and nothing ever decides to set it. It is the default waiting reason the kubelet writes for any container it has not created yet (kubelet_pods.go), so a pod stuck there means the scheduler has finished and the kubelet has not: a volume will not mount, the pod network could not be set up, or the image is still downloading.

The container status has no message to give you, because the container does not exist. The answer is in the events.

The fastest check

kubectl events --for pod/<pod>

LAST SEEN             TYPE      REASON        OBJECT    MESSAGE
4m15s                 Normal    Scheduled     Pod/web   Successfully assigned k21cc/web to glance-blog-control-plane
5s (x10 over 4m15s)   Warning   FailedMount   Pod/web   MountVolume.SetUp failed for volume "cfg" : configmap "app-config" not found
5s (x10 over 4m15s)   Warning   FailedMount   Pod/web   MountVolume.SetUp failed for volume "creds" : secret "app-creds" not found

The reason column sorts you into one of three buckets: FailedMount or FailedAttachVolume is storage, FailedCreatePodSandBox is the network plugin, and a Pulling with no Pulled after it is an image download still in progress.

When you have several stuck pods, one condition splits them without reading any events. PodReadyToStartContainers turns True only once the pod sandbox exists and has an IP:

kubectl get pods -o custom-columns='NAME:.metadata.name,STATUS:.status.containerStatuses[0].state.waiting.reason,SANDBOX:.status.conditions[?(@.type=="PodReadyToStartContainers")].status'

NAME         STATUS              SANDBOX
builder      ContainerCreating   True
db-pinned    ContainerCreating   False
db-unbound   ContainerCreating   False
web          ContainerCreating   False

False means the kubelet never got as far as the sandbox: volumes or CNI. True means everything before the image pull worked, and builder here is simply downloading a 522 MB image. The condition has been on by default since 1.29 and is GA in 1.37 (kube_features.go).

What the kubelet does between Scheduled and Running

The steps run in a fixed order, and each one has to finish before the next starts. That order is what makes the event reason diagnostic: it tells you which step you are stuck on and, by implication, which steps already succeeded.

Zoom

Two things in that sequence surprise people. Volumes come first — the kubelet waits for every volume in WaitForAttachAndMount before it asks the runtime for anything (kubelet.go, runtime sync at L2231). And a failed pull does not leave you in ContainerCreating: it replaces the reason with ErrImagePull. Only a slow pull stays here.

1. A volume will not mount

This is the common case. The events above are a ConfigMap and a Secret mounted as volumes, neither of which exists. The kubelet retries each mount with backoff, which is where the x10 over 4m15s comes from.

The same missing ConfigMap referenced from env instead of volumes does not produce this at all. It gets past the mount step, fails at config generation, and shows up as a named error in the container status — covered in CreateContainerConfigError and CreateContainerError. Same mistake, different step, different place to look.

The fix is to create the object, in the pod’s namespace. You do not need to delete the pod:

kubectl create configmap app-config --from-literal=mode=prod
kubectl create secret generic app-creds --from-literal=token=x

On 1.36.1 the pod picked both up on its own and moved to Pulling just under two minutes later, because the mount retry had backed off by then. If the volume is genuinely optional, say so with optional: true under configMap: or secret:, and the kubelet mounts an empty directory instead of waiting.

Reading “Unable to attach or mount volumes: unmounted volumes=[…]”

This is the message people search for, and on current versions it means something more specific than it used to. Two pods referencing claims that are not usable, verified on 1.36.1:

Warning  FailedMount  kubelet  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[], failed to process volumes=[data]: error processing PVC k21cc/data-missing: failed to fetch PVC from API server: persistentvolumeclaims "data-missing" not found
Warning  FailedMount  kubelet  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[], failed to process volumes=[data]: error processing PVC k21cc/data-unbound: PVC is not bound

Read it as three lists and a reason:

A volume in the third list is a configuration problem, not a slow disk: the claim does not exist, is not bound, or conflicts with another pod. A volume only in the second list is an attach that has not completed, which is a storage backend or CSI question.

The old timeout variant is gone. Older threads quote the same message ending : timed out waiting for the condition. The kubelet waits 2m3s for volumes (volume_manager.go), and up to 1.27 it recorded an event when that wait expired. From 1.28 it skips the event on a plain timeout and emits it only when there is a hard error (kubelet.go release-1.27 against release-1.37). So on a modern cluster, a pod whose volume is merely slow shows only the per-volume MountVolume.SetUp failed or attach events — the missing-ConfigMap pod above sat for over six minutes without ever producing the summary line.

The useful event is often on another object. The PVC is not bound pod says nothing about why. The claim does:

kubectl events --for pvc/data-unbound

LAST SEEN              TYPE      REASON               OBJECT                               MESSAGE
3m53s (x26 over 10m)   Warning   ProvisioningFailed   PersistentVolumeClaim/data-unbound   storageclass.storage.k8s.io "fast-ssd" not found

Attach failures follow the same pattern from the other direction: they are raised by the attach/detach controller in kube-controller-manager, not the kubelet, with reason FailedAttachVolume. The classic one is a ReadWriteOnce disk still attached to the node the previous pod ran on. Its message begins Multi-Attach error for volume on most clusters and Waiting for detach for volume from 1.35.9, 1.36.4 and 1.37.0 — a patch-release change, so two clusters on the same minor can print different text (reconciler.go v1.36.1, v1.36.4). Both forms are covered in Multi-Attach error for volume. The cluster-wide view of what is attached where:

kubectl get volumeattachment

One caveat about the two PVC examples: I pinned those pods with spec.nodeName to get them past the scheduler. A normally scheduled pod with a missing claim never reaches the kubelet — it stays Pending with 0/1 nodes are available: persistentvolumeclaim "data-missing" not found, which is covered in pod Pending and FailedScheduling. You see these in ContainerCreating when something bypasses the scheduler, or when the claim changes after the pod was bound.

2. The pod network could not be set up

Once volumes are mounted, the kubelet asks the runtime for a sandbox, and the runtime calls the CNI plugin to give it an interface and an IP. On a separate kind cluster I moved the ptp plugin binary out of /opt/cni/bin:

Warning  FailedCreatePodSandBox  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "13ec93afaad379611d7a7d175c1b360f6b2d732b8bc515b46fe257000487fb44": plugin type="ptp" failed (add): failed to find plugin "ptp" in path [/opt/cni/bin]

The sandbox ID changes on every retry, roughly every twelve seconds, because the kubelet tears the failed sandbox down and starts again. The tail names the plugin and what it could not do. Your CNI’s wording will differ; the tail after failed to setup network for sandbox "…": is still the part that names the problem.

What catches people is that the node stayed Ready throughout. The CNI config file was still in place — only the binary it names was gone — and nothing on the node object reflected that every new pod on it would fail. Only the pods did. Check the network plugin’s own pods on the affected node:

kubectl get pods -n kube-system -o wide --field-selector spec.nodeName=<node>

When the binary came back, the stuck pod created its sandbox on the next retry and moved to Pulling without being touched. Pods that already had a sandbox kept running throughout, which is why a CNI failure looks like “new pods on one node are broken” rather than an outage.

3. The image is still pulling

SANDBOX is True, and the events end at Pulling:

LAST SEEN   TYPE     REASON      OBJECT        MESSAGE
83s         Normal   Scheduled   Pod/builder   Successfully assigned k21cc/builder to glance-blog-control-plane
83s         Normal   Pulling     Pod/builder   Pulling image "rust:1-bookworm"

Nothing is wrong. There is no progress event, so for a large image the pod looks exactly the same at second 10 and at minute 4. It finished like this:

Normal   Pulled      Pod/builder   Successfully pulled image "rust:1-bookworm" in 4m35.45s (4m35.45s including waiting). Image size: 522048568 bytes.

The including waiting figure is the one to read, because the kubelet pulls one image at a time by default (serializeImagePulls, default true, types.go). A 3 MB image queued behind a 400 MB one on the same node:

Normal   Pulled   Pod/small2   Successfully pulled image "alpine:3.19" in 15.368s (2m53.333s including waiting). Image size: 3369000 bytes.

Three minutes of ContainerCreating for a tiny image, entirely spent in someone else’s queue. Check what your nodes are set to:

kubectl get --raw "/api/v1/nodes/<node>/proxy/configz" | jq '.kubeletconfig | {serializeImagePulls, maxParallelImagePulls}'

{
  "serializeImagePulls": true,
  "maxParallelImagePulls": null
}

Setting serializeImagePulls: false with a maxParallelImagePulls limit in the kubelet configuration lets pulls overlap. Smaller images and pre-pulling onto nodes help more. If the pull fails rather than drags, the pod leaves this state for ErrImagePull — see ImagePullBackOff and ErrImagePull for reading those messages.

4. New in v1.37: two pods, one volume, conflicting SELinux labels

This only applies to nodes with SELinux enforcing. The kind nodes used for everything above do not run SELinux, so this section is source-checked against release-1.37 and the upstream announcement, not reproduced.

v1.37 turns on the SELinuxMount feature gate by default and graduates it to GA (kube_features.go, PR #139956, listed in the v1.37 changelog under Urgent Upgrade Notes). Instead of the runtime relabelling every file on a volume, the kubelet mounts eligible volumes with -o context=<label>, so one mount carries one label. Eligible means in-tree FC and iSCSI, and CSI drivers whose CSIDriver sets seLinuxMount: true:

kubectl get csidriver -o custom-columns='NAME:.metadata.name,SELINUXMOUNT:.spec.seLinuxMount'

The consequence, in the words of the Kubernetes blog: two setups that worked under recursive relabelling — pods with different labels on different subPaths of one volume, and a privileged and an unprivileged pod sharing a volume — no longer do, and “one of these Pods will be stuck in ContainerCreating until the other Pod is terminated”.

On the stuck pod, the conflict is detected while the kubelet builds its desired volume state (desired_state_of_world.go), recorded as a pod error (desired_state_of_world_populator.go), and surfaces through the same Unable to attach or mount volumes: … failed to process volumes=[…] event as the PVC cases above, with a tail built from conflicting SELinux labels of volume %s: %q and %q. The volume is in the third list, and the reason names two labels but not the other pod.

To find the other pod, there is an opt-in selinux-warning-controller in kube-controller-manager, still disabled by default in 1.37 (core.go), enabled with --controllers=*,selinux-warning-controller. It puts an event on both pods with reason SELinuxLabelConflict (or SELinuxChangePolicyConflict), naming the peer when it is in the same namespace and hiding it when it is not (conflict.go). Two details that will cost you time otherwise: those events are type Normal, not Warning (selinux_warning_controller.go), so kubectl events --types=Warning hides them; and the controller reports conflicts even when the pods are on different nodes today, so the warning may predate the failure. It also exports selinux_warning_controller_selinux_volume_conflict with both pods as labels, which is the way to audit a 1.36 cluster before upgrading.

The per-pod opt-out is a stable field, present on 1.36:

spec:
  securityContext:
    seLinuxChangePolicy: Recursive

Recursive restores relabelling by the runtime for that pod’s volumes, which is slower on large volumes and allows the sharing. The better fix is usually the one the upstream post suggests: ask whether the privileged pod needs to be privileged.

Preventing it

• Look at events before you delete the pod. Every cause here recovers on its own once the cause is fixed, and deleting the pod throws away the events that named it.
• Mount config with optional: true where absence is legitimate. A missing optional ConfigMap costs nothing; a required one blocks the pod indefinitely.
• Watch pull time, not pull failure. Big images do not fail, they just make every new pod on a fresh node slow, and with serialised pulls they make other people’s pods slow too.
• Alert on PodReadyToStartContainers=False older than a few minutes. It catches storage and CNI problems in one condition, including on nodes that still report Ready.
• Run the SELinux warning controller before you upgrade to 1.37 if any node enforces SELinux. It is the only thing that tells you about the conflict before a pod gets stuck.