Your GPU pod is CrashLoopBackOff and Kubernetes will not say the device failed

When a GPU fails underneath a running pod, Kubernetes tells you the container exited. It does not tell you the device is why. The pod enters CrashLoopBackOff, the exit code is whatever CUDA returned, and every field kubectl shows you describes the container rather than the hardware it was given.

This is not an oversight in your setup. It is a known gap, and the KEP that closes it names the symptom in its own motivation: pods enter CrashLoopBackOff without any indication that the assigned device is the root cause.

The fastest check, where the field is enabled

There is now a field for it, though nothing surfaces it by default:

kubectl get pod <pod> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.allocatedResourcesStatus}{"\n"}{end}'

On a 1.36 cluster the field exists in the core API and reports per-resource health:

kubectl explain pod.status.containerStatuses.allocatedResourcesStatus.resources.health

Health of the resource. can be one of:
 - Healthy: operates as normal
 - Unhealthy: reported unhealthy. We consider this a temporary health issue
              since we do not have a mechanism today to distinguish
              temporary and permanent issues.
 - Unknown: The status cannot be determined.
            For example, Device Plugin got unregistered and hasn't been
            re-registered since.

Unknown is the one to read carefully. It does not mean the device is fine and unmonitored — it commonly means the device plugin itself went away, which is a different fault with a different fix from a bad card.

The feature gate is ResourceHealthStatus, and on the 1.36.1 cluster used here it reports as beta and enabled:

kubectl get --raw /metrics | grep ResourceHealthStatus

kubernetes_feature_enabled{name="ResourceHealthStatus",stage="BETA"} 1

On a managed cluster, check rather than assume — a beta gate can be disabled by the provider, and the field is simply absent when it is.

Why the failure is invisible without it

A device plugin advertises devices to the kubelet and reports their health on a stream. The kubelet allocates a device to a container at admission and passes the device nodes into the container. After that, from the pod’s point of view, the device is just a file.

Zoom

The consequence is the whole problem: the failure and the report take different paths. The plugin knows the GPU has fallen off the bus. The container knows a CUDA call failed and exits non-zero. The kubelet restarts the container as it would for any non-zero exit, backs off, and the pod’s status carries a generic CrashLoopBackOff. Nothing in the pod’s status ever mentioned the device, because before this field there was nowhere to put it.

Node capacity does eventually reflect it — an unhealthy device is withdrawn from the advertised count — but that shows up on the Node, not on the pod, and only after the plugin notices.

Diagnosing it today

Work outward from the pod, because everything useful is outside it.

1. Is the resource still advertised on the node?

kubectl get node <node> -o jsonpath='{.status.allocatable}{"\n"}'
kubectl describe node <node> | grep -A5 Allocatable

A node that advertised nvidia.com/gpu: 8 yesterday and 7 today has lost a card. This single check answers the question more often than anything done to the pod.

2. Is the device plugin healthy?

kubectl -n kube-system get pods -l name=nvidia-device-plugin-ds -o wide
kubectl -n kube-system logs <plugin-pod> --tail=50

The plugin is a DaemonSet pod like any other, and it has its own restart count. A plugin that is itself crash-looping withdraws every device on the node, which presents as your workload being unschedulable or repeatedly killed, and the node looking fine.

3. What did the container actually say?

kubectl logs <pod> --previous

--previous is the important flag: the current container has just started and its logs will not contain the failure. Device errors usually arrive as an initialisation failure from the runtime library rather than as a message about hardware, so read the first few lines rather than the last.

4. Are the events saying something the pod status is not?

kubectl get events --field-selector involvedObject.name=<pod> --sort-by=.lastTimestamp

The states a device moves through

Zoom

The distinction the API itself flags is worth repeating: there is currently no way to express permanently unhealthy. Every unhealthy report is treated as potentially temporary, which is why a genuinely dead card can produce a pod that keeps being scheduled onto the same node and keeps failing. Until that changes, a node with confirmed bad hardware needs cordoning by you, not by the cluster.

What changes when resource health is populated

Once the field is populated in your environment, the diagnosis becomes one command instead of four, and more usefully it becomes machine-readable. An alert can distinguish “this application crashes” from “this hardware failed”, which today requires a human to correlate a node’s allocatable count against a pod’s restart count.

It also changes what a client can show you. Pod status has, until now, had no concept of the resources a container holds being in a bad state — which is why every dashboard in the category, ours included, shows the same CrashLoopBackOff for both cases. A status algorithm that reads the STATUS column faithfully gains a genuinely new input here, not a cosmetic one.

The same field covers Dynamic Resource Allocation claims, where the resource name takes the form claim:<claim_name>/<request> rather than a plain resource name. That matters because DRA is where accelerator scheduling is heading, and a diagnosis that only understands the device-plugin naming will stop working.

Practical advice until then

• Alert on node allocatable for accelerator resources. A drop is a hardware event and it is visible immediately, unlike anything on the pod.
• Treat the device plugin DaemonSet as a critical workload. Its restarts are more informative about GPU incidents than your workload’s restarts.
• Always read --previous logs. For device failures the current container’s logs are almost always useless, since the interesting thing happened before the restart.
• Do not raise memory limits at a GPU crash loop. It is the reflex from OOMKilled and exit code 137 and it wastes the first hour. Check lastState.terminated.reason first — if it is Error rather than OOMKilled, memory is not involved.
• Cordon a node with a confirmed bad device. Kubernetes will keep trying, because as far as the API is concerned the problem might be temporary.

For the general case of a container that will not stay up, including the causes that have nothing to do with hardware, see how to fix CrashLoopBackOff.