
troubleshooting
HPA TARGETS <unknown> and FailedGetResourceMetric: the five messages behind it
HPA TARGETS stuck at cpu: <unknown>/50% with FailedGetResourceMetric? The end of the ScalingActive message names one of five causes. Each one, with its fix.
<unknown> in the TARGETS column of kubectl get hpa means the HPA controller
tried to compute that metric on its last pass and could not. It sets the
ScalingActive condition to False with reason FailedGetResourceMetric,
records two Warning events, and stops scaling the workload in either direction
until the metric comes back.
Why it could not is always in the last clause of the condition message. There are five possible endings. Two of them clear up on their own within a minute, and three need a fix.
The fastest check
kubectl get hpa
kubectl describe hpa web
On a Kubernetes 1.37.0 cluster with no metrics-server:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web cpu: <unknown>/50% 2 5 2 77s
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 50%
Min replicas: 2
Max replicas: 5
Deployment pods: 2 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 2s (x6 over 77s) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Warning FailedComputeMetricsReplicas 2s (x6 over 77s) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Read the ScalingActive message from the right-hand end:
| The message ends with | What it means | Clears by itself? |
|---|---|---|
the server could not find the requested resource (get pods.metrics.k8s.io) | Nothing serves the metrics API. No metrics-server | No |
the server is currently unable to handle the request (get pods.metrics.k8s.io) | The metrics API is registered, but its backend is unavailable | No |
no metrics returned from resource metrics API | metrics-server answered with no pods at all | Yes, about 30s after new pods start |
did not receive metrics for targeted pods (pods might be unready) | Metrics exist, but only for pods the HPA sets aside | Usually, during a rollout |
missing request for cpu in container X of Pod Y | Container X has no CPU request | No |
kubectl top pods splits the problem in half. For the first two it fails with
error: Metrics API not available. For the last one it works perfectly: the
usage numbers are there, and the HPA has nothing to divide them by.
The TARGETS cell itself changed format in Kubernetes 1.30, when #122804 added the resource name. Search results carry both forms:
| Cluster version | TARGETS for a CPU utilization target with no metric |
|---|---|
| 1.29 and earlier | <unknown>/50% |
| 1.30 to 1.37 | cpu: <unknown>/50% |
The cell is rendered by the API server, not by kubectl, so the format follows the
cluster’s version. formatHPAMetrics (pkg/printers/internalversion/printers.go,
line 2500 on release-1.36, 2543 on release-1.37) prints <unknown> whenever
the status entry for that metric is empty.
Every output here is from kind clusters: variant 1 on 1.37.0, the rest on 1.36.1
with metrics-server v0.9.0. Source references are to release-1.36 and
release-1.37, where the HPA code involved has the same line numbers unless
stated.
How the HPA gets a number
The HPA controller runs inside kube-controller-manager and visits each HPA every
15 seconds (--horizontal-pod-autoscaler-sync-period, defaulted in
pkg/controller/podautoscaler/config/v1alpha1/defaults.go line 41). For a
Resource metric with a utilization target, GetResourceReplicas in
pkg/controller/podautoscaler/replica_calculator.go (line 80) does this:
- List pod metrics from
metrics.k8s.io/v1beta1with the target’s label selector. The API server does not serve that group itself. Its aggregation layer forwards the request to the Service named by thev1beta1.metrics.k8s.ioAPIService, normally metrics-server, which answers from memory with the latest rate it computed by scraping each kubelet. - List the pods matching the same selector, from the controller’s cache.
- Set some pods aside (
groupPods, line 422): terminating and Failed pods, Pending pods, and for CPU, pods whose samples might be start-up noise. - Sum the requests of every container in each pod, plus native sidecars
(
calculatePodRequestsFromContainers, line 516). - Divide total usage by total requests across the pods with metrics
(
metrics/utilization.goline 49). It is a ratio of sums, not an average of each pod’s percentage, so a pod with a large request weighs more. - Compare with the target. Within a tolerance of 0.1 either side of the
target (
defaults.goline 53), nothing changes.
The five messages are the five ways to fall out of that sequence before step 5.
What happens next is identical for all five. getUnableComputeReplicaCountCondition
(horizontal.go line 1194 on release-1.37, 1143 on release-1.36) records a
Warning event with reason FailedGetResourceMetric and sets ScalingActive to
False with the message prefixed the HPA was unable to compute the replica count:. The caller wraps the same error in invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: and records a second
Warning, FailedComputeMetricsReplicas (line 946 on release-1.37, 895 on
release-1.36). The status is written with an empty entry for the metric, and
that is the <unknown>.
Then it stops: no replica count, so no scaling. Three consequences:
- Desired replicas is not a decision. A new HPA shows
2 current / 0 desiredbecause it has never computed one. An HPA that worked before keeps showing its last value. minReplicasandmaxReplicasstill apply. Those checks run before any metric is fetched (horizontal.golines 925–934 onrelease-1.37), so a workload below the minimum is still raised to it.- With several metrics, one broken metric blocks only scale-down. If the
metrics that do work ask for more replicas, the HPA scales up; if the result
would be fewer, it refuses (
horizontal.goline 419 onrelease-1.37, 354 onrelease-1.36). A memory target that silently fails next to a working CPU target gives a Deployment that can grow and never shrink.
The tolerance is also configurable per HPA now, through
spec.behavior.scaleUp.tolerance and spec.behavior.scaleDown.tolerance. The
HPAConfigurableTolerance gate is beta and on by default from 1.35
(pkg/features/kube_features.go line 1498 on release-1.36) and GA, locked on,
in 1.37 (line 1647). None of that matters while the metric is <unknown>.
1. the server could not find the requested resource: no metrics API
The HPA asked the API server for pods.metrics.k8s.io, and the API server has
never heard of it. No APIService is registered for the group, so the request gets
a 404, which client-go spells the server could not find the requested resource
(staging/src/k8s.io/apimachinery/pkg/api/errors/errors.go line 450).
kubectl top pods
kubectl get apiservice v1beta1.metrics.k8s.io
error: Metrics API not available
Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io" not found
This is the default on kind and kubeadm clusters, which do not install a metrics pipeline. Whether a managed cluster ships one depends on the provider and on how the cluster was created, so check rather than assume.
Fix it by installing metrics-server from
its release manifest,
or anything else that registers v1beta1.metrics.k8s.io, such as
prometheus-adapter. On
kind the kubelets serve self-signed certificates; the metrics-server on the 1.36
cluster used here runs with --kubelet-insecure-tls, which is for a laptop and
nothing else. There is no need to recreate the HPA. It retries every sync and
picks up the metric once the API answers.
2. the server is currently unable to handle the request: registered but unavailable
The APIService exists, but the aggregation layer has marked it unavailable. When
that is the case it does not even try the backend: handler_proxy.go in
staging/src/k8s.io/kube-aggregator/pkg/apiserver/ returns a 503 straight away
(lines 123–125), and a 503 reads as the server is currently unable to handle the request (errors.go line 481).
Scaling the 1.36 cluster’s metrics-server to zero for a minute reproduces it:
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl get apiservice v1beta1.metrics.k8s.io -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
kubectl describe hpa web
NAME SERVICE AVAILABLE AGE
v1beta1.metrics.k8s.io kube-system/metrics-server False (MissingEndpoints) 35d
Available False MissingEndpoints endpointslices for service/metrics-server in "kube-system" have no addresses with port name "https"
Deployment pods: 2 current / 2 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
The HPA message is the same whatever went wrong, so the diagnosis is on the
APIService. The availability controller in
kube-aggregator/pkg/controllers/status/remote/remote_available_controller.go
sets the reason:
| APIService reason | What to look at |
|---|---|
ServiceNotFound | The Service the APIService points at was deleted. Usually a half-uninstalled metrics-server |
MissingEndpoints | No ready metrics-server pod. Its pods, their events and logs |
FailedDiscoveryCheck | The API server could not reach the pod. The message has the URL and the network error |
FailedDiscoveryCheck is the same path an admission webhook takes, from the API
server into a pod on a node, and it breaks the same ways: firewalls between the
control plane and the nodes, and NetworkPolicies that do not admit the API server.
Fix it by fixing whatever the APIService reason names. Once the condition goes
back to True, kubectl top nodes works again and the HPA recovers on its next
sync.
3. no metrics returned from resource metrics API: nothing to report yet
metrics-server answered, with an empty list (metrics/client.go line 74). The
most common reason is harmless. Creating a Deployment and its HPA together on
1.36.1 and checking every five seconds:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web cpu: <unknown>/50% 2 5 0 5s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web cpu: <unknown>/50% 2 5 2 26s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web cpu: 0%/50% 2 5 2 31s
metrics-server serves a pod only once it has two samples to turn into a CPU rate
(pkg/storage/pod.go lines 51–59 in metrics-server v0.9.0). The official manifest
scrapes every 15 seconds; the binary’s own default is 60
(cmd/metrics-server/app/options/options.go line 113), which stretches the gap
to as much as two minutes. How that window is computed is covered in
why kubectl top disagrees with your dashboard.
The trap is that the events outlive the problem. A minute later the same HPA:
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 50s horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
The condition is the current state. The events are history, kept for an hour by
default. If ScalingActive is True, the warnings are from before it was.
If the message persists on pods that have been running for minutes,
metrics-server has no data for any of them. Pods it has no samples for are
skipped rather than reported as errors, so when every pod of the workload sits on
nodes whose kubelets it cannot scrape, this empty list is all the HPA sees. The
metrics-server logs say which: Failed to scrape node, with the node name.
4. did not receive metrics for targeted pods (pods might be unready)
Here metrics-server did return data, but every pod it covered was one the HPA sets
aside (replica_calculator.go line 97). Adding a missing request to a Deployment,
which rolls it, produced this on 1.36.1:
Warning FailedGetResourceMetric 57s (x2 over 72s) horizontal-pod-autoscaler failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
The old pods had metrics but were terminating; the new ones were running but had no samples yet. It cleared on the next sync after the new pods’ first rate.
The case that does not clear is readiness. For CPU, groupPods treats a pod as
unready, and drops its sample, if in its first five minutes
(--horizontal-pod-autoscaler-cpu-initialization-period, defaults.go line 47)
it is not Ready or its sample predates it becoming Ready. After that, it is
unready only if it is not Ready and its readiness has not changed since its first
30 seconds (--horizontal-pod-autoscaler-initial-readiness-delay, line 50): in
effect, it has never been Ready. A Deployment whose readiness probe never passes
therefore has no CPU samples the HPA trusts. Fix the readiness first; see
Pod is Running but 0/1 READY.
When only some pods are missing, there is no error. The HPA fills the gaps
conservatively (replica_calculator.go lines 121–142): on a scale-down, a pod
with no metric is counted as using its full request; on a scale-up, it and any
unready pod count as using nothing. If those assumptions would reverse the
direction, it keeps the current count. That is why an HPA can scale more slowly
than the raw numbers suggest during a rollout.
5. missing request for cpu in container X of Pod Y
metrics-server is fine. The HPA has usage and nothing to divide it by. A Deployment whose app container requests 100m and whose log shipper requests nothing, on 1.36.1:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
api Deployment/api cpu: <unknown>/50% 2 5 2 75s
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container log-shipper of Pod api-886bc9c65-4zbkt
kubectl top pods --containers
POD NAME CPU(cores) MEMORY(bytes)
api-886bc9c65-4zbkt api 0m 11Mi
api-886bc9c65-4zbkt log-shipper 0m 0Mi
api-886bc9c65-mptcb api 0m 11Mi
api-886bc9c65-mptcb log-shipper 0m 0Mi
The names are not quoted in this message
(replica_calculator.go line 529), which matters if you are searching logs for
it. The rules that decide which containers count:
- One container in one pod fails the whole HPA. The error returns as soon as any container lacks the request; there is no partial answer.
- Native sidecars count.
calculatePodRequestsFromContainersappends every init container withrestartPolicy: Always(line 519). A proxy sidecar with no request producedmissing request for cpu in container proxy of Pod with-sidecar-6d5bf5b7ff-tgsxm. How sidecars add to a pod’s request elsewhere is in the order of your init containers changes your Pod’s CPU request. - Ordinary init containers do not. The same pod with a request-less
migrateinit container instead reportedValidMetricFound. - Injected sidecars count. A mesh proxy is in the pod, not in your manifest. Read the container name in the message, not the YAML you applied.
- A limit on its own is enough. Defaulting copies a container’s limits into
missing requests (
pkg/apis/core/v1/defaults.goline 170 onrelease-1.37, 165 onrelease-1.36), so only a container with neither is a problem. - Every pod the selector matches counts. Requests are summed over the whole
pod list (line 100 passes it to
calculateRequests), including pods of another workload that happen to carry the same labels. The pod name in the message says whose it is.
Fix it in one of three ways, all tested on 1.36.1.
Give the container a request. The Deployment rolls, and the HPA recovered within a minute of the rollout finishing:
kubectl set resources deployment api -c log-shipper --requests=cpu=10m
Or set a pod-level request. When spec.resources.requests is set on the pod, the
HPA uses it and ignores the containers (calculateRequests, line 475). That has
worked since 1.34, with the PodLevelResources gate beta and on by default
(still beta in 1.37, kube_features.go line 1965). A pod requesting cpu: 200m
at pod level, with two containers that request nothing, scaled normally.
Or measure only the container you care about, with a ContainerResource metric
(GA since 1.30, when the HPAContainerMetrics gate was locked on):
metrics:
- type: ContainerResource
containerResource:
name: cpu
container: api
target:
type: Utilization
averageUtilization: 50
That HPA reported ValidMetricFound ... from cpu container resource utilization
with the log shipper still unrequested. Replace the old HPA rather than adding a
second one. With both pointing at the same Deployment, the new one went to
ScalingActive False with reason AmbiguousSelector: pods by selector app=api are controlled by multiple HPAs: [k33-hpa/api-container k33-hpa/api].
For injected sidecars, set the request through the mesh’s own configuration. The injector decides what the sidecar gets, every time a pod is created.
Preventing it
- Make the metrics API part of cluster bootstrap.
kubectl get apiservice v1beta1.metrics.k8s.ioshowingTrueis a one-line smoke test, and it covers both of the first two causes. - Alert on the condition, not the events. Causes 3 and 4 fire whenever
pods are new, on every deploy and rollout. kube-state-metrics exposes
kube_horizontalpodautoscaler_status_conditionwithconditionandstatuslabels (metric reference); alert whenScalingActivehas beenfalsefor several minutes. - Put a request on every container of an autoscaled workload, including the ones you did not write. A LimitRange with a default request catches containers that arrive without one, at pod creation.
- Prefer
ContainerResourcewhen a pod carries sidecars. It scales on the container whose load you mean, and a sidecar’s request can no longer break or dilute the number.
An HPA stuck on <unknown> says why in its ScalingActive condition, one screen away from the workload. KubeGlance is a native Kubernetes client for iPhone and iPad, with a full Mac app on the same core: its Autoscaling view lists each HPA and KEDA target, flags metrics it cannot read, and shows the conditions word for word.
Get KubeGlanceThe Kubernetes dashboard that fits in your pocket
KubeGlance is a native Kubernetes client for iPhone and iPad — the real dashboard, not a companion — with a full Mac app on the same core. Pods, workloads, logs and events, straight from your kubeconfig.
Download KubeGlance

