KubeGlanceDownload
HPA TARGETS <unknown> and FailedGetResourceMetric: the five messages behind it

troubleshooting

HPA TARGETS <unknown> and FailedGetResourceMetric: the five messages behind it

HPA TARGETS stuck at cpu: <unknown>/50% with FailedGetResourceMetric? The end of the ScalingActive message names one of five causes. Each one, with its fix.

· 11 min read

<unknown> in the TARGETS column of kubectl get hpa means the HPA controller tried to compute that metric on its last pass and could not. It sets the ScalingActive condition to False with reason FailedGetResourceMetric, records two Warning events, and stops scaling the workload in either direction until the metric comes back.

Why it could not is always in the last clause of the condition message. There are five possible endings. Two of them clear up on their own within a minute, and three need a fix.

The fastest check

kubectl get hpa
kubectl describe hpa web

On a Kubernetes 1.37.0 cluster with no metrics-server:

NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: <unknown>/50%   2         5         2          77s
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          2
Max replicas:                                          5
Deployment pods:                                       2 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedGetResourceMetric       2s (x6 over 77s)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
  Warning  FailedComputeMetricsReplicas  2s (x6 over 77s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

Read the ScalingActive message from the right-hand end:

The message ends withWhat it meansClears by itself?
the server could not find the requested resource (get pods.metrics.k8s.io)Nothing serves the metrics API. No metrics-serverNo
the server is currently unable to handle the request (get pods.metrics.k8s.io)The metrics API is registered, but its backend is unavailableNo
no metrics returned from resource metrics APImetrics-server answered with no pods at allYes, about 30s after new pods start
did not receive metrics for targeted pods (pods might be unready)Metrics exist, but only for pods the HPA sets asideUsually, during a rollout
missing request for cpu in container X of Pod YContainer X has no CPU requestNo

kubectl top pods splits the problem in half. For the first two it fails with error: Metrics API not available. For the last one it works perfectly: the usage numbers are there, and the HPA has nothing to divide them by.

The TARGETS cell itself changed format in Kubernetes 1.30, when #122804 added the resource name. Search results carry both forms:

Cluster versionTARGETS for a CPU utilization target with no metric
1.29 and earlier<unknown>/50%
1.30 to 1.37cpu: <unknown>/50%

The cell is rendered by the API server, not by kubectl, so the format follows the cluster’s version. formatHPAMetrics (pkg/printers/internalversion/printers.go, line 2500 on release-1.36, 2543 on release-1.37) prints <unknown> whenever the status entry for that metric is empty.

Every output here is from kind clusters: variant 1 on 1.37.0, the rest on 1.36.1 with metrics-server v0.9.0. Source references are to release-1.36 and release-1.37, where the HPA code involved has the same line numbers unless stated.

How the HPA gets a number

The HPA controller runs inside kube-controller-manager and visits each HPA every 15 seconds (--horizontal-pod-autoscaler-sync-period, defaulted in pkg/controller/podautoscaler/config/v1alpha1/defaults.go line 41). For a Resource metric with a utilization target, GetResourceReplicas in pkg/controller/podautoscaler/replica_calculator.go (line 80) does this:

  1. List pod metrics from metrics.k8s.io/v1beta1 with the target’s label selector. The API server does not serve that group itself. Its aggregation layer forwards the request to the Service named by the v1beta1.metrics.k8s.io APIService, normally metrics-server, which answers from memory with the latest rate it computed by scraping each kubelet.
  2. List the pods matching the same selector, from the controller’s cache.
  3. Set some pods aside (groupPods, line 422): terminating and Failed pods, Pending pods, and for CPU, pods whose samples might be start-up noise.
  4. Sum the requests of every container in each pod, plus native sidecars (calculatePodRequestsFromContainers, line 516).
  5. Divide total usage by total requests across the pods with metrics (metrics/utilization.go line 49). It is a ratio of sums, not an average of each pod’s percentage, so a pod with a large request weighs more.
  6. Compare with the target. Within a tolerance of 0.1 either side of the target (defaults.go line 53), nothing changes.

The five messages are the five ways to fall out of that sequence before step 5.

Each message is produced at a different step. Where it comes from tells you whose problem it is: the API server's, metrics-server's, the pods' readiness, or your pod spec.

What happens next is identical for all five. getUnableComputeReplicaCountCondition (horizontal.go line 1194 on release-1.37, 1143 on release-1.36) records a Warning event with reason FailedGetResourceMetric and sets ScalingActive to False with the message prefixed the HPA was unable to compute the replica count:. The caller wraps the same error in invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: and records a second Warning, FailedComputeMetricsReplicas (line 946 on release-1.37, 895 on release-1.36). The status is written with an empty entry for the metric, and that is the <unknown>.

Then it stops: no replica count, so no scaling. Three consequences:

  • Desired replicas is not a decision. A new HPA shows 2 current / 0 desired because it has never computed one. An HPA that worked before keeps showing its last value.
  • minReplicas and maxReplicas still apply. Those checks run before any metric is fetched (horizontal.go lines 925–934 on release-1.37), so a workload below the minimum is still raised to it.
  • With several metrics, one broken metric blocks only scale-down. If the metrics that do work ask for more replicas, the HPA scales up; if the result would be fewer, it refuses (horizontal.go line 419 on release-1.37, 354 on release-1.36). A memory target that silently fails next to a working CPU target gives a Deployment that can grow and never shrink.

The tolerance is also configurable per HPA now, through spec.behavior.scaleUp.tolerance and spec.behavior.scaleDown.tolerance. The HPAConfigurableTolerance gate is beta and on by default from 1.35 (pkg/features/kube_features.go line 1498 on release-1.36) and GA, locked on, in 1.37 (line 1647). None of that matters while the metric is <unknown>.

1. the server could not find the requested resource: no metrics API

The HPA asked the API server for pods.metrics.k8s.io, and the API server has never heard of it. No APIService is registered for the group, so the request gets a 404, which client-go spells the server could not find the requested resource (staging/src/k8s.io/apimachinery/pkg/api/errors/errors.go line 450).

kubectl top pods
kubectl get apiservice v1beta1.metrics.k8s.io
error: Metrics API not available
Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io" not found

This is the default on kind and kubeadm clusters, which do not install a metrics pipeline. Whether a managed cluster ships one depends on the provider and on how the cluster was created, so check rather than assume.

Fix it by installing metrics-server from its release manifest, or anything else that registers v1beta1.metrics.k8s.io, such as prometheus-adapter. On kind the kubelets serve self-signed certificates; the metrics-server on the 1.36 cluster used here runs with --kubelet-insecure-tls, which is for a laptop and nothing else. There is no need to recreate the HPA. It retries every sync and picks up the metric once the API answers.

2. the server is currently unable to handle the request: registered but unavailable

The APIService exists, but the aggregation layer has marked it unavailable. When that is the case it does not even try the backend: handler_proxy.go in staging/src/k8s.io/kube-aggregator/pkg/apiserver/ returns a 503 straight away (lines 123–125), and a 503 reads as the server is currently unable to handle the request (errors.go line 481).

Scaling the 1.36 cluster’s metrics-server to zero for a minute reproduces it:

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl get apiservice v1beta1.metrics.k8s.io -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
kubectl describe hpa web
NAME                     SERVICE                      AVAILABLE                  AGE
v1beta1.metrics.k8s.io   kube-system/metrics-server   False (MissingEndpoints)   35d
Available	False	MissingEndpoints	endpointslices for service/metrics-server in "kube-system" have no addresses with port name "https"
Deployment pods:                                       2 current / 2 desired
Conditions:
  Type            Status  Reason                   Message
  ----            ------  ------                   -------
  AbleToScale     True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive   False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
  ScalingLimited  False   DesiredWithinRange       the desired count is within the acceptable range

The HPA message is the same whatever went wrong, so the diagnosis is on the APIService. The availability controller in kube-aggregator/pkg/controllers/status/remote/remote_available_controller.go sets the reason:

APIService reasonWhat to look at
ServiceNotFoundThe Service the APIService points at was deleted. Usually a half-uninstalled metrics-server
MissingEndpointsNo ready metrics-server pod. Its pods, their events and logs
FailedDiscoveryCheckThe API server could not reach the pod. The message has the URL and the network error

FailedDiscoveryCheck is the same path an admission webhook takes, from the API server into a pod on a node, and it breaks the same ways: firewalls between the control plane and the nodes, and NetworkPolicies that do not admit the API server.

Fix it by fixing whatever the APIService reason names. Once the condition goes back to True, kubectl top nodes works again and the HPA recovers on its next sync.

3. no metrics returned from resource metrics API: nothing to report yet

metrics-server answered, with an empty list (metrics/client.go line 74). The most common reason is harmless. Creating a Deployment and its HPA together on 1.36.1 and checking every five seconds:

NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: <unknown>/50%   2         5         0          5s
NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: <unknown>/50%   2         5         2          26s
NAME   REFERENCE        TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: 0%/50%   2         5         2          31s

metrics-server serves a pod only once it has two samples to turn into a CPU rate (pkg/storage/pod.go lines 51–59 in metrics-server v0.9.0). The official manifest scrapes every 15 seconds; the binary’s own default is 60 (cmd/metrics-server/app/options/options.go line 113), which stretches the gap to as much as two minutes. How that window is computed is covered in why kubectl top disagrees with your dashboard.

The trap is that the events outlive the problem. A minute later the same HPA:

Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range
Events:
  Type     Reason                        Age   From                       Message
  ----     ------                        ----  ----                       -------
  Warning  FailedGetResourceMetric       50s   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

The condition is the current state. The events are history, kept for an hour by default. If ScalingActive is True, the warnings are from before it was.

If the message persists on pods that have been running for minutes, metrics-server has no data for any of them. Pods it has no samples for are skipped rather than reported as errors, so when every pod of the workload sits on nodes whose kubelets it cannot scrape, this empty list is all the HPA sees. The metrics-server logs say which: Failed to scrape node, with the node name.

4. did not receive metrics for targeted pods (pods might be unready)

Here metrics-server did return data, but every pod it covered was one the HPA sets aside (replica_calculator.go line 97). Adding a missing request to a Deployment, which rolls it, produced this on 1.36.1:

  Warning  FailedGetResourceMetric       57s (x2 over 72s)  horizontal-pod-autoscaler  failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

The old pods had metrics but were terminating; the new ones were running but had no samples yet. It cleared on the next sync after the new pods’ first rate.

The case that does not clear is readiness. For CPU, groupPods treats a pod as unready, and drops its sample, if in its first five minutes (--horizontal-pod-autoscaler-cpu-initialization-period, defaults.go line 47) it is not Ready or its sample predates it becoming Ready. After that, it is unready only if it is not Ready and its readiness has not changed since its first 30 seconds (--horizontal-pod-autoscaler-initial-readiness-delay, line 50): in effect, it has never been Ready. A Deployment whose readiness probe never passes therefore has no CPU samples the HPA trusts. Fix the readiness first; see Pod is Running but 0/1 READY.

When only some pods are missing, there is no error. The HPA fills the gaps conservatively (replica_calculator.go lines 121–142): on a scale-down, a pod with no metric is counted as using its full request; on a scale-up, it and any unready pod count as using nothing. If those assumptions would reverse the direction, it keeps the current count. That is why an HPA can scale more slowly than the raw numbers suggest during a rollout.

5. missing request for cpu in container X of Pod Y

metrics-server is fine. The HPA has usage and nothing to divide it by. A Deployment whose app container requests 100m and whose log shipper requests nothing, on 1.36.1:

NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
api    Deployment/api   cpu: <unknown>/50%   2         5         2          75s
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container log-shipper of Pod api-886bc9c65-4zbkt
kubectl top pods --containers
POD                    NAME          CPU(cores)   MEMORY(bytes)   
api-886bc9c65-4zbkt    api           0m           11Mi            
api-886bc9c65-4zbkt    log-shipper   0m           0Mi             
api-886bc9c65-mptcb    api           0m           11Mi            
api-886bc9c65-mptcb    log-shipper   0m           0Mi             

The names are not quoted in this message (replica_calculator.go line 529), which matters if you are searching logs for it. The rules that decide which containers count:

  • One container in one pod fails the whole HPA. The error returns as soon as any container lacks the request; there is no partial answer.
  • Native sidecars count. calculatePodRequestsFromContainers appends every init container with restartPolicy: Always (line 519). A proxy sidecar with no request produced missing request for cpu in container proxy of Pod with-sidecar-6d5bf5b7ff-tgsxm. How sidecars add to a pod’s request elsewhere is in the order of your init containers changes your Pod’s CPU request.
  • Ordinary init containers do not. The same pod with a request-less migrate init container instead reported ValidMetricFound.
  • Injected sidecars count. A mesh proxy is in the pod, not in your manifest. Read the container name in the message, not the YAML you applied.
  • A limit on its own is enough. Defaulting copies a container’s limits into missing requests (pkg/apis/core/v1/defaults.go line 170 on release-1.37, 165 on release-1.36), so only a container with neither is a problem.
  • Every pod the selector matches counts. Requests are summed over the whole pod list (line 100 passes it to calculateRequests), including pods of another workload that happen to carry the same labels. The pod name in the message says whose it is.

Fix it in one of three ways, all tested on 1.36.1.

Give the container a request. The Deployment rolls, and the HPA recovered within a minute of the rollout finishing:

kubectl set resources deployment api -c log-shipper --requests=cpu=10m

Or set a pod-level request. When spec.resources.requests is set on the pod, the HPA uses it and ignores the containers (calculateRequests, line 475). That has worked since 1.34, with the PodLevelResources gate beta and on by default (still beta in 1.37, kube_features.go line 1965). A pod requesting cpu: 200m at pod level, with two containers that request nothing, scaled normally.

Or measure only the container you care about, with a ContainerResource metric (GA since 1.30, when the HPAContainerMetrics gate was locked on):

  metrics:
  - type: ContainerResource
    containerResource:
      name: cpu
      container: api
      target:
        type: Utilization
        averageUtilization: 50

That HPA reported ValidMetricFound ... from cpu container resource utilization with the log shipper still unrequested. Replace the old HPA rather than adding a second one. With both pointing at the same Deployment, the new one went to ScalingActive False with reason AmbiguousSelector: pods by selector app=api are controlled by multiple HPAs: [k33-hpa/api-container k33-hpa/api].

For injected sidecars, set the request through the mesh’s own configuration. The injector decides what the sidecar gets, every time a pod is created.

Only the tail of the message matters. The first two are the metrics pipeline, the middle two are timing or readiness, the last is your pod spec.

Preventing it

  • Make the metrics API part of cluster bootstrap. kubectl get apiservice v1beta1.metrics.k8s.io showing True is a one-line smoke test, and it covers both of the first two causes.
  • Alert on the condition, not the events. Causes 3 and 4 fire whenever pods are new, on every deploy and rollout. kube-state-metrics exposes kube_horizontalpodautoscaler_status_condition with condition and status labels (metric reference); alert when ScalingActive has been false for several minutes.
  • Put a request on every container of an autoscaled workload, including the ones you did not write. A LimitRange with a default request catches containers that arrive without one, at pod creation.
  • Prefer ContainerResource when a pod carries sidecars. It scales on the container whose load you mean, and a sidecar’s request can no longer break or dilute the number.

An HPA stuck on <unknown> says why in its ScalingActive condition, one screen away from the workload. KubeGlance is a native Kubernetes client for iPhone and iPad, with a full Mac app on the same core: its Autoscaling view lists each HPA and KEDA target, flags metrics it cannot read, and shows the conditions word for word.

Get KubeGlance
#hpa #autoscaling #metrics-server #requests #troubleshooting

The Kubernetes dashboard that fits in your pocket

KubeGlance is a native Kubernetes client for iPhone and iPad — the real dashboard, not a companion — with a full Mac app on the same core. Pods, workloads, logs and events, straight from your kubeconfig.

Download KubeGlance