Why kubectl top disagrees with your dashboard

kubectl top and your Grafana panel are reading the same counters and computing different things from them. Three differences account for nearly every discrepancy:

• kubectl top prints a rate averaged over a window whose length is whatever the last two scrapes happened to be apart, not a fixed interval.
• It rounds to whole millicores and whole mebibytes.
• metrics-server keeps only the latest sample, so nothing about the past is available to it and nothing you see is a maximum.

None of that is a bug. It is what the metrics API is for — it exists to feed the autoscaler, and the autoscaler wants one recent number per pod, cheaply.

The window is in the API, and it is not 15 seconds

kubectl top hides it, but the underlying resource says exactly what it measured:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods/<pod>"

{
  "kind": "PodMetrics",
  "metadata": { "name": "cpudemo", "namespace": "blogtest" },
  "timestamp": "2026-08-25T11:14:38Z",
  "window": "22.299s",
  "containers": [
    { "name": "app", "usage": { "cpu": "0", "memory": "152Ki" } }
  ]
}

Two fields worth staring at. window is 22.299s on a cluster whose scrape interval is nominally 15 seconds — the window is the actual gap between the two cumulative CPU samples the rate was computed from, and it drifts with load. And timestamp is in the past: you are always looking at a measurement taken before you asked.

So “CPU over the last 15 seconds” is already an approximation of what the number means, and comparing it against a Prometheus rate(...[5m]) compares a ~20-second average with a 5-minute one. During a spike, the short window is higher. During a lull, it is lower. They will agree only when the workload is flat.

Zoom

The rounding hides small numbers entirely

The same pod above reports 152Ki in the raw API. kubectl top prints:

NAME      NAME   CPU(cores)   MEMORY(bytes)
cpudemo   app    0m           0Mi

0Mi, for a container using 152 kibibytes, and 0m for CPU. For sidecars, small utilities and anything idle, kubectl top will show a column of zeros while your dashboard shows real numbers. Neither is lying.

Working set is not RSS, and it is not what you think either

The memory number is container_memory_working_set_bytes: total cgroup memory minus inactive file-backed pages. It deliberately includes active page cache, because that is the memory the kernel would have to reclaim under pressure — and reclaiming it is not free.

Consequences that surprise people:

• A container that reads a lot of files shows working set well above the RSS its own runtime reports. The process is not leaking; the kernel is caching.
• Working set is also the number the OOM killer’s threshold relates to, which is why it is the right one to watch even though it is not “your program’s memory”.
• A JVM reporting a 400 MB heap in a container whose working set is 900 MB is normal. Heap is a subset of RSS, which is a subset of working set.

If your dashboard panel is built on container_memory_rss, it will always read lower than kubectl top, permanently, and neither number is wrong.

The flat graph is not evidence

metrics-server holds the most recent sample per pod and discards the rest. There is no history in the metrics API at all — kubectl top cannot show you yesterday, and neither can anything else built solely on metrics.k8s.io.

That matters most in the case where you actually need it. A container that allocates hard and dies in 200 milliseconds does so entirely between two scrapes. The sample before and the sample after both look fine, and if your dashboard is also scraping every 15 or 30 seconds, so does it.

Zoom

A flat memory graph up to the moment of an OOMKilled is therefore not a reason to doubt the kill — it is the expected picture, and it is why OOMKilled and exit code 137 tells you to trust lastState.terminated.reason over the graph.

Throttling is not in there at all

This is the gap that costs the most time. metrics.k8s.io carries CPU and memory usage and nothing else — you can see the whole schema in the raw response above. There is no field for CFS throttling, so no amount of looking at kubectl top will tell you a container is being throttled, and a throttled container’s CPU usage looks low precisely because it is being prevented from using CPU.

The counter lives in cAdvisor and is only reachable through Prometheus or an equivalent:

rate(container_cpu_cfs_throttled_periods_total[5m])
/
rate(container_cpu_cfs_periods_total[5m])

A service that is slow with low CPU usage and no obvious cause is throttled far more often than it is anything else. Why the limit does that on an idle node is covered in CPU requests vs limits.

When top disagrees, which do you believe

That last one is worth its own note, because it is the most common kubectl top failure and it is not a metrics problem:

kubectl get apiservice v1beta1.metrics.k8s.io

A healthy one reports AVAILABLE True. Anything else means the metrics deployment is the fault, not your cluster’s monitoring. On clusters with self-signed kubelet certificates — kind and several managed distributions — metrics-server needs --kubelet-insecure-tls or a properly signed kubelet serving certificate, and without it the APIService never becomes available.

Use each for what it is

• kubectl top for “what is happening now”, on one pod, right now. It is fast, it needs no extra tooling, and it is accurate for what it claims.
• Prometheus or an equivalent for anything involving time, which includes every capacity decision, every limit you are about to set, and every “was it memory”.
• lastState.terminated.reason for what killed something. No metric will ever answer that as well as the field that records it.

For the column that reports those terminations, see what RESTARTS 3 (5m ago) is actually counting.