PVC stuck in Pending: the four messages, and the one that is not an error

A PersistentVolumeClaim in Pending has not been bound to a PersistentVolume yet. The persistentvolume-controller always says why, as an event on the claim, and there are only four things it says.

One of them, waiting for first consumer to be created before binding, is not a fault at all. It is the most debugged non-problem in Kubernetes storage.

The fastest check

kubectl describe pvc <pvc>

The Events section is the answer. On a fresh claim against a WaitForFirstConsumer StorageClass, with no pod using it yet:

Name:          data
Namespace:     demo
StorageClass:  standard
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age   From                         Message
  ----    ------                ----  ----                         -------
  Normal  WaitForFirstConsumer  6s    persistentvolume-controller  waiting for first consumer to be created before binding

Match the Reason column:

Events expire. The API server keeps them for one hour by default (--event-ttl), so a claim that has been Pending since yesterday may show Events: <none>. The annotations tell you the same story and do not expire:

kubectl get pvc <pvc> -o jsonpath='{.spec.storageClassName}{"\t"}{.metadata.annotations.volume\.kubernetes\.io/storage-provisioner}{"\t"}{.metadata.annotations.volume\.kubernetes\.io/selected-node}{"\n"}'

Three claims from the reproductions below, one per line:

fast	ebs.csi.aws.com	
standard		
standard	rancher.io/local-path	glance-blog-k27-control-plane

Empty class: the FailedBinding case. A class but no provisioner annotation: still waiting for a consumer or for the pod to schedule. A provisioner but no selected node: an Immediate class waiting on the provisioner. Both set: the scheduler has picked a node, and the provisioner is the thing to look at.

Every output in this article is from a kind cluster on Kubernetes 1.36.1, and the message strings are source-checked on release-1.37.

How a claim gets bound

The claim is shepherded by two components that never talk directly. The persistentvolume-controller in kube-controller-manager owns the claim. The scheduler owns the pod. Under WaitForFirstConsumer they coordinate through a single annotation on the PVC.

Zoom

The detail that explains most confusion: under WaitForFirstConsumer the controller does nothing on its own. It waits until the scheduler has chosen a node for a pod that uses the claim and written volume.kubernetes.io/selected-node on the PVC. Only then does it tag the claim with the provisioner name and emit ExternalProvisioning. In the source, the switch in syncUnboundClaim checks IsDelayBindingProvisioning, which is nothing more than “does the claim carry the selected-node annotation”.

The scheduler, meanwhile, holds the pod in its PreBind step until the claim is bound, for up to bindTimeoutSeconds, which defaults to 600 (pkg/scheduler/apis/config/v1/defaults.go on release-1.36, line 196).

A healthy dynamic provision emits three of the four messages in order. This is a claim that went through all of it and succeeded:

Events:
  Type    Reason                 Age                From                                                                                                Message
  ----    ------                 ----               ----                                                                                                -------
  Normal  WaitForFirstConsumer   49s                persistentvolume-controller                                                                         waiting for first consumer to be created before binding
  Normal  WaitForPodScheduled    20s (x2 over 35s)  persistentvolume-controller                                                                         waiting for pod app to be scheduled
  Normal  ExternalProvisioning   15s (x2 over 15s)  persistentvolume-controller                                                                         Waiting for a volume to be created either by the external provisioner 'rancher.io/local-path' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  Provisioning           15s                rancher.io/local-path_local-path-provisioner-855c7b7774-n82vc_ccc413c4-67fb-42a0-b587-e26add26b098  External provisioner is provisioning volume for claim "demo/data"
  Normal  ProvisioningSucceeded  12s                rancher.io/local-path_local-path-provisioner-855c7b7774-n82vc_ccc413c4-67fb-42a0-b587-e26add26b098  Successfully provisioned volume pvc-8c7a2883-8b51-44f2-8ce6-58033b56acba

So seeing one of these messages is not the diagnosis. Seeing it last, with nothing after it, is.

1. no persistent volumes available for this claim and no storage class is set

The claim has no storageClassName, the cluster has no default StorageClass to fill it in, and no pre-created PV matches. Nothing is going to provision a volume, so the controller just keeps looking for a static PV.

With the default annotation removed from the only class:

NAME       PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
fast       ebs.csi.aws.com         Delete          Immediate              false                  112s
standard   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  3m39s

StorageClass:  
Status:        Pending
...
Events:
  Type    Reason         Age               From                         Message
  ----    ------         ----              ----                         -------
  Normal  FailedBinding  9s (x2 over 20s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

kubectl get sc marks the default with (default). No marker, no default. This is common on self-built clusters and after someone uninstalls the storage add-on that shipped with the distribution.

Fix it by marking a class as default:

kubectl annotate storageclass standard storageclass.kubernetes.io/is-default-class=true

Since 1.28, when the retroactive default StorageClass assignment went GA (#118102), the controller then fills in storageClassName on existing claims that had none. You do not have to recreate the PVC. Twenty seconds after annotating, the same claim:

Events:
  Type    Reason                Age                From                         Message
  ----    ------                ----               ----                         -------
  Normal  FailedBinding         33s (x2 over 44s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
  Normal  WaitForFirstConsumer  3s (x2 over 18s)   persistentvolume-controller  waiting for first consumer to be created before binding

One exception, and it catches people: storageClassName: "" is not “no class”. It is an explicit request for no dynamic provisioning, and a default class is never applied to it. A claim written that way produces exactly the same message even with a default class present. It binds only to a PV that also has an empty class, sufficient capacity and matching access modes. Created by hand, such a PV binds within seconds:

NAME     STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
static   Bound    demo-static-pv   1Gi        RWO                           <unset>                 30s

If you meant dynamic provisioning, delete the "" from the manifest; a pending claim’s spec is immutable, so that means recreating the PVC.

2. waiting for first consumer to be created before binding

This is the one that is not an error. The StorageClass has volumeBindingMode: WaitForFirstConsumer, and no pod references the claim yet. The controller is deliberately not provisioning, because it does not yet know which node, and therefore which zone, the volume needs to be in.

It is the mode of the default class in kind (local-path). A PVC created ahead of its Deployment, or by a Helm chart whose workload is scaled to zero, will sit here indefinitely and correctly.

Fix it by creating the pod. If you genuinely need the volume to exist first, use a class with volumeBindingMode: Immediate, and accept that the volume may land in a zone your pod cannot reach.

3. waiting for pod <name> to be scheduled

Same WaitForFirstConsumer mode, but now a pod references the claim. The reason flips from WaitForFirstConsumer to WaitForPodScheduled, and the message names the pod. If several unscheduled pods reference the same claim, the source prints waiting for pods a,b to be scheduled instead.

Used By:       app
Events:
  Type    Reason                Age               From                         Message
  ----    ------                ----              ----                         -------
  Normal  WaitForFirstConsumer  30s               persistentvolume-controller  waiting for first consumer to be created before binding
  Normal  WaitForPodScheduled   1s (x2 over 16s)  persistentvolume-controller  waiting for pod app to be scheduled

The PVC is not the problem here. It is waiting on the pod, and the pod is waiting on a node. Describe the pod it names:

kubectl describe pod app

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  20s   default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

That pod has a nodeSelector no node satisfies. Labelling the node was enough: the scheduler picked it, wrote selected-node, the provisioner created the volume and the claim went Bound within fifteen seconds. Whatever the scheduler reports, work through it as a scheduling problem; the reasons are covered in Pod Pending and FailedScheduling.

4. ExternalProvisioning: the provisioner has not answered

The controller has done its part. It has written volume.kubernetes.io/storage-provisioner on the claim and is waiting for whatever is registered under that name to create a PV. It does not check whether anything by that name exists.

A class naming a provisioner that is not installed, on 1.36.1:

Events:
  Type    Reason                Age                From                         Message
  ----    ------                ----               ----                         -------
  Normal  ExternalProvisioning  12s (x4 over 30s)  persistentvolume-controller  Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

The message is identical when the provisioner is installed but not running. Scaling kind’s local-path-provisioner to zero replicas and creating a claim plus pod produced the same line with 'rancher.io/local-path', and the pod sat in Pending with Events: <none>: it was held in the scheduler’s PreBind step, not rejected by it. Scaling the provisioner back to one bound the claim within seconds.

Two wordings of the same message

Search results show two versions of this event, and people paste whichever their cluster printed. On Kubernetes 1.27 and earlier:

waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator

From 1.28 onwards:

Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

The change is #118262, merged May 2023 for v1.28. The old form is in pv_controller.go on release-1.24 through release-1.27; the new form is on every branch from release-1.28, and is unchanged on release-1.35, release-1.36 and release-1.37 (line 1837 of pkg/controller/volume/persistentvolume/pv_controller.go on the last two). Same event reason, same meaning; only the text moved.

Fix it by finding the provisioner. For CSI drivers, that is a controller Deployment or StatefulSet running csi-provisioner as a sidecar; for others, a standalone pod:

kubectl get pods -A | grep -i provisioner

local-path-storage   local-path-provisioner-855c7b7774-kj4vw                 1/1     Running   0          2m58s

No pod at all means it was never installed, or the name in the StorageClass is wrong. Check the spelling against what the driver registers: a CSI driver usually creates a CSIDriver object with its name, so kubectl get csidrivers returning No resources found while your class says ebs.csi.aws.com settles it. Not every provisioner is CSI; local-path is not, and has no CSIDriver entry.

If the pod exists but is crashing or has no permissions, its logs are the next stop, not the PVC.

A wrong provisioner name cannot be patched out:

The StorageClass "fast" is invalid: provisioner: Invalid value: "rancher.io/local-path": field is immutable

Nor can the claim be moved to another class:

The PersistentVolumeClaim "fast-data" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims

Recreate the StorageClass with the right name, then delete and recreate the PVC.

PVC stuck in Terminating

The opposite problem has one cause almost every time. Every claim carries the kubernetes.io/pvc-protection finalizer, visible in the first describe above. The PVC protection controller removes it only once no pod uses the claim.

kubectl delete pvc data --wait=false
kubectl get pvc data

NAME   STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data   Terminating   pvc-8c7a2883-8b51-44f2-8ce6-58033b56acba   1Gi        RWO            standard       <unset>                 4m18s

kubectl describe pvc names the user in the Used By: line:

VolumeMode:    Filesystem
Used By:       app

Delete the pod (or scale down whatever owns it) and the claim goes immediately.

The case that surprises people: a completed pod still counts. In the protection controller on release-1.36, podUsesPVCForDeletion treats any scheduled pod that references the claim as a user, without checking whether it has finished. A restartPolicy: Never pod that ran true and exited:

NAME   READY   STATUS      RESTARTS   AGE
once   0/1     Completed   0          6s

still held its PVC in Terminating, with Used By: once. Finished Job and CronJob pods left lying around are the usual culprits. When Used By is hard to read, list every pod that mounts the claim:

kubectl get pods -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="data") | .metadata.name + "\t" + .status.phase'

app	Running

Deleting once released scratch straight away. Do not reach for removing the finalizer: it exists so a volume is not deleted from under a pod that may still be writing to it. The PV has its own, kubernetes.io/pv-protection, for the same reason. The general version of this finalizer problem is covered in Pod stuck in Terminating.

Preventing it

• Keep exactly one default StorageClass, and check for it in cluster bootstrap. kubectl get sc with no (default) is a one-line smoke test.
• Treat WaitForFirstConsumer as expected. Alert on claims whose last event is ExternalProvisioning or WaitForPodScheduled for more than a few minutes, not on Pending itself.
• Pin provisioner names from the driver’s own docs into your StorageClass manifests, and install the driver before the class. Both fields you would want to fix later are immutable.
• Clean up finished Job pods with ttlSecondsAfterFinished if they mount claims you intend to delete.

When the claim binds but the pod then fails to attach it on a second node, that is a different message; see Multi-Attach error for volume.