OOMKilled and exit code 137: what actually happened

OOMKilled almost never means the node ran out of memory. It means your container exceeded the limit you gave it, and the kernel killed it on a machine that may have had gigabytes free.

That is why raising the node size so often changes nothing, and why the graph you are staring at shows plenty of headroom at the moment of death. The limit is a cgroup boundary around one container. Nothing outside it is relevant.

The fastest check

kubectl get pod <pod> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

api     OOMKilled       137

Two fields, and you need both. reason tells you what killed it; exitCode alone does not, because 137 is produced by every SIGKILL, not just an OOM.

Across a whole namespace:

kubectl get pods -o custom-columns='NAME:.metadata.name,REASON:.status.containerStatuses[*].lastState.terminated.reason,QOS:.status.qosClass'

Exit code 137 is not a synonym for OOMKilled

137 is 128 + 9 — the process received SIGKILL. Something killed it without asking. There are three common somethings, and two of them are not memory:

If reason is Error rather than OOMKilled, memory is not your problem and no amount of raising limits will help. Look at the probe configuration instead — that path is covered in how to fix CrashLoopBackOff, where it is one of the more commonly misdiagnosed causes.

Three different kills that look the same

Zoom

1. Container cgroup OOM — the common one

The container exceeded resources.limits.memory. The kernel kills the largest process in that cgroup. The pod stays on its node, the container restarts, the restart count goes up, and the pod’s phase never changes.

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].restartCount}{"\n"}'

This is the one that produces reason: OOMKilled.

2. Node-pressure eviction — the one that moves your pod

The node ran low on memory and the kubelet evicted whole pods to reclaim it. This is a different mechanism entirely: the pod is deleted, not restarted, and it comes back on another node — or nowhere, if nothing else can take it.

kubectl get pod <pod> -o jsonpath='{.status.reason}{"\n"}'

That prints Evicted, and kubectl describe carries a message naming the resource under pressure. The order the kubelet evicts in is decided by QoS class:

Guaranteed is not a setting you switch on. It is what you get when requests and limits match exactly on CPU and memory for every container in the pod, init containers included. One mismatched sidecar downgrades the entire pod to Burstable, which is a genuinely common way to lose protection you thought you had:

kubectl get pod <pod> -o jsonpath='{.status.qosClass}{"\n"}'

3. System OOM — the one that kills the wrong thing

The node itself exhausted memory faster than the kubelet’s eviction loop could react, and the kernel’s OOM killer chose a victim by score. It may pick something completely unrelated to the pod that caused the pressure. These show up in the node’s kernel log rather than in Kubernetes, which is why they can be invisible from kubectl entirely.

Reserving memory for the system properly — kube-reserved, system-reserved, and a hard eviction threshold — is what keeps the kubelet ahead of the kernel. If you are seeing system OOMs, that configuration is the fix, not the workload.

Why the numbers never match

The most common report is some version of “the app uses 300 MB and I set a limit of 512 MB, and it still gets OOMKilled.” Three reasons, and they compound.

Page cache counts against your limit. The cgroup accounts for file-backed pages your container caused to be read, not just anonymous memory. A service that reads a lot of files can sit well above its RSS in cgroup terms. Under pressure the kernel reclaims that cache before killing anything, but it counts until it does.

Zoom

kubectl top shows working set, and it lags. metrics-server scrapes on an interval — 15 seconds by default, sometimes more. An allocation spike that kills your container in 200 milliseconds happens entirely between two samples. If the graph is flat right up to the kill, that is not evidence the memory was fine.

kubectl top pod <pod> --containers

The kill is against the peak, not the average. One request that decodes a large payload, one garbage collection cycle that has not run yet, one connection pool that grew during a retry storm. Averages will never show you the number that mattered.

Runtime-specific causes worth checking first

A large share of OOMKilled incidents are a runtime that does not know it is in a container, or knows and was told the wrong thing.

JVM. UseContainerSupport has been on by default since JDK 10, so a modern JVM reads the cgroup limit — but the default MaxRAMPercentage leaves the heap at a fraction of the limit, and off-heap memory (metaspace, thread stacks, direct buffers) is not counted in it at all. A heap sized at 100% of the container limit is guaranteed to be killed. Set -XX:MaxRAMPercentage=75 and leave the rest for everything the heap does not include.

Node.js. Node 20 and later size the V8 heap from the cgroup limit rather than from the host, so the classic “512 MB container, heap sized for a 64 GB machine” failure is mostly historical — it still bites on older runtimes, and it bit everyone in cgroup v2 environments until libuv learned to read them (nodejs/node#47259). What is still true on every version is that --max-old-space-size bounds the old space only. Buffers, native addons and the rest of the process live outside it, so a heap ceiling set at the container limit still gets killed. Set it explicitly, and set it below the limit.

Go. Go 1.25 made GOMAXPROCS container-aware — on Linux the runtime reads the cgroup CPU bandwidth limit instead of counting host cores, and re-reads it periodically in case the limit changes. That is worth having, and it takes some pressure off memory as a side effect: fewer Ps means fewer per-P allocation caches and less allocated between collections. But it is a CPU fix, and the headline is easy to misread as “Go understands containers now”. Memory has no equivalent. With GOMEMLIMIT unset the collector still targets a ratio of live heap, so a burst of allocation raises the target and the process grows straight through the limit. The proposal to derive GOMEMLIMIT from the cgroup memory limit (golang/go#75164) is still open — it missed 1.26 and 1.27, and the argument is over how much headroom to leave by default. So set it yourself: GOMEMLIMIT at roughly 90% of limits.memory and the collector works harder instead of dying. This is the single highest-value change for a Go service that OOMs under load, and it needs no code change. Read the limit from the cgroup in your entrypoint rather than hardcoding it, and the value stays right when something resizes the pod under you.

Fixing it

Size the limit from observed peak, not from average. If you have a metrics stack, container_memory_working_set_bytes at the 99th percentile over a couple of weeks is the number, plus headroom. If you do not, raise the limit until the kills stop and then investigate whether the number is reasonable — do not leave it there without asking.

Set requests equal to limits for anything you cannot afford to lose. That is what buys Guaranteed and moves you to the back of the eviction queue.

Do not set a limit far above the request on memory. Memory is incompressible — unlike CPU, a container cannot be throttled down once it has allocated. A pod that requests 256 MB and is allowed 4 GB will be scheduled as though it needs 256 MB and can then take a node down. Overcommitting CPU is normal practice. Overcommitting memory is how nodes die.

Resize instead of restarting, on 1.33 and later. In-place pod resize was beta and on by default in 1.33 and went stable in 1.35, so on a recent cluster you can raise a limit without recreating the pod. Decreasing a memory limit in place was prohibited until it was permitted in the 1.35 stable release. On anything older, changing resources still means a new pod.

Preventing it

• Alert on restartCount increasing. A container OOMing every twenty minutes and passing its readiness probe in between is invisible to an uptime check and will be reported to you by a user.
• Watch the ratio, not the absolute. Working set as a fraction of the limit is the signal. 400 MB means nothing on its own; 96% of the limit means you have days.
• Treat a Guaranteed pod that silently became Burstable as a regression. It usually happens when someone adds a sidecar and does not give it limits.
• Check the QoS class in review, not in the incident. It is one field and it decides who dies first.

Where a pod never came back after an eviction, the follow-on problem is usually placement rather than memory, and the scheduler will tell you exactly why in 0/3 nodes are available.