
troubleshooting
Internal error occurred: failed calling webhook: context deadline exceeded, and every other ending
failed calling webhook ... context deadline exceeded: the API server could not get an answer from an admission webhook. What each ending means and the fix.
Your object was never judged. Before storing it, the API server has to send it
to an admission webhook, and that call failed: the API server could not reach
the webhook, could not trust it, or did not get an answer within
timeoutSeconds. Because the webhook’s failurePolicy is Fail, which is the
default, the API server refuses the whole request and reports it as its own
failure: HTTP 500, Internal error occurred.
The part that tells you what went wrong is the text after the URL. For
context deadline exceeded, it means the API server waited the full timeout and
got nothing back:
time kubectl apply -f cm.yaml
Error from server (InternalError): error when creating "cm.yaml": Internal error occurred: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=10s": context deadline exceeded
kubectl apply -f cm.yaml 0.07s user 0.03s system 0% cpu 10.088 total
Ten seconds exactly: the default timeout, spent before the ConfigMap was even
looked at. Every output here is from a kind cluster on Kubernetes 1.36.1, with a
small TLS webhook of my own in front of ConfigMaps. The code paths are
source-checked on release-1.36 and release-1.37.
The fastest check
1. Read the ending. The same code path produces all of these. Each was reproduced on 1.36.1 except the second, which is quoted from source:
| Text after the URL | What happened |
|---|---|
context deadline exceeded | No reply within timeoutSeconds. The webhook is slow, or the packets never arrive |
no endpoints available for service "k32-hook" | No ready webhook pod, on an API server that routes to endpoints |
dial tcp 10.96.38.100:443: connect: connection refused | No ready webhook pod, on an API server that routes to the ClusterIP (the default) |
service "k32-hook-typo" not found | The configuration names a Service that does not exist |
no service port 8443 found for service "k32-hook" | clientConfig.service.port is not a port on the Service. It wants the Service port, not the container port |
tls: failed to verify certificate: x509: certificate signed by unknown authority | caBundle is wrong or empty |
tls: failed to verify certificate: x509: certificate is valid for k32-hook.default.svc, not k32-hook.k32-webhook.svc | The serving certificate was issued for another name |
One more comes from the same place with a slightly different shape, because
the webhook did answer:
failed calling webhook "…": received invalid webhook response: expected response.uid="a50c77b4-…", got "not-the-request-uid".
And one that is not this problem at all:
admission webhook "…" denied the request: … means the call worked and the
webhook said no. Read its reason and fix your object.
2. Find the configuration. The name in quotes is a webhook name, not the name of the configuration that contains it. List both, with where they send traffic and what happens when that fails:
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o custom-columns='CONFIG:.metadata.name,WEBHOOK:.webhooks[*].name,SERVICE:.webhooks[*].clientConfig.service.name,NAMESPACE:.webhooks[*].clientConfig.service.namespace,POLICY:.webhooks[*].failurePolicy,TIMEOUT:.webhooks[*].timeoutSeconds'
CONFIG WEBHOOK SERVICE NAMESPACE POLICY TIMEOUT
k32-selflock k32-pods.k32.example.com k32-hook k32-webhook Fail 10
k32-defaults k32-defaults.k32.example.com k32-hook k32-webhook Ignore 5
3. Check the Service has somewhere to send it:
kubectl -n k32-webhook get endpointslices -l kubernetes.io/service-name=k32-hook
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
k32-hook-wnks8 IPv4 <unset> <unset> 5m44s
<unset> means no ready pod. Go to the webhook’s Deployment. If there are
endpoints and you still get context deadline exceeded, the problem is between
the API server and the pod, or inside the webhook.
What the API server is doing
The webhook is called by kube-apiserver itself, from wherever kube-apiserver runs. Not by kubectl, and not from inside a pod. That one fact explains most of this error.
Where it sits in the request. A create runs mutating admission first
(endpoints/handlers/create.go line 203), then the resource’s own validation
(rest.BeforeCreate, registry/generic/registry/store.go line 509 on
release-1.36), then validating admission (line 515). Mutating webhooks are
called one after another in a loop (mutating/dispatcher.go line 127);
validating webhooks are called in parallel, one goroutine each
(validating/dispatcher.go line 130). Slow mutating webhooks add their waits
together; slow validating webhooks overlap.
How it finds the pod. The URL in the message, https://k32-hook.k32-webhook.svc:443,
is built from clientConfig.service (pkg/util/webhook/client.go line 187 on
release-1.36), but the API server never looks that name up in DNS. Its dialer
swaps in an address from a service resolver (line 204; on 1.37 a wrapping
round tripper does the same job by default). Which resolver depends
on one flag, in buildServiceResolver (cmd/kube-apiserver/app/server.go
lines 283–300):
- Default: the Service’s ClusterIP. The connection then goes through
kube-proxy on the API server’s own node, which is why an empty Service shows
up as
connection refused: kube-proxy installs a REJECT rule for a Service with no endpoints (pkg/proxy/iptables/proxier.goline 910; the nftables proxier does the same). --enable-aggregator-routing=true: a ready endpoint picked from the EndpointSlices, dialled directly. No ready endpoint givesno endpoints available for service(pkg/util/proxy/proxy.golines 82 and 126 on release-1.36, 78 and 149 on release-1.37). k3s turns this flag on unless its egress selector is disabled (pkg/daemons/control/server.goline 239, default modeagentinpkg/cli/cmds/server.go), and the string turns up verbatim in reports from Rancher (rancher/rancher#54739), EKS (aws-load-balancer-controller#2336) and OpenShift (Red Hat solution 7051156). Same cause asconnection refused, different wording. In 1.37 this path also load-balances across endpoints instead of reusing one connection (#139237, feature gateWebhookRoundTripLoadBalancing, beta and on by default).
What it trusts. TLS is verified against caBundle only, with the server
name forced to <service>.<namespace>.svc (client.go lines 155 and 194 on
release-1.36). The certificate must be signed by that bundle and valid for that
exact name.
How long it waits. timeoutSeconds defaults to 10
(pkg/apis/admissionregistration/v1/defaults.go lines 50 and 74) and must be
between 1 and 30 (validation.go line 388):
The ValidatingWebhookConfiguration "k32-policy" is invalid: webhooks[0].timeoutSeconds: Invalid value: 31: the timeout value must be between 1 and 30 seconds
The dispatcher puts that timeout on the call’s context (dispatcher.go line
279) and also passes it to the webhook as the ?timeout=10s in the URL, so a
well-behaved webhook knows its budget. Setting timeoutSeconds: 3 against the
same stuck backend:
Error from server (InternalError): error when creating "cm.yaml": Internal error occurred: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=3s": context deadline exceeded
kubectl apply -f cm.yaml 0.07s user 0.03s system 3% cpu 3.089 total
Why it says “Internal error”. The transport failure becomes
failed calling webhook "<name>": failed to call webhook: <Go error>
(pkg/util/webhook/error.go line 36, validating/dispatcher.go line 311).
With failurePolicy: Fail, the default (defaults.go lines 33 and 57), it is
wrapped in NewInternalError (validating/dispatcher.go line 218,
mutating/dispatcher.go line 226), which prefixes Internal error occurred:
and sets status 500 (apimachinery/pkg/api/errors/errors.go line 395). The
server is telling you that it failed, not that your object is bad. The line
numbers are the same on release-1.37 unless noted. The failed to call webhook:
middle part was added in 1.23; older API servers print the Go error straight
after the webhook name.
Why it happens, and the fix for each
1. No ready webhook pods
Rule this out first. The webhook’s Deployment is scaled down, crash-looping, not yet
ready after an install or upgrade, or its only node is gone. The ending is
connection refused or no endpoints available for service, and it fails
instantly rather than after ten seconds. Scaled to zero:
Error from server (InternalError): error when creating "cm.yaml": Internal error occurred: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=10s": dial tcp 10.96.38.100:443: connect: connection refused
kubectl apply -f cm.yaml 0.05s user 0.02s system 106% cpu 0.064 total
Behind a Deployment you will not see this at the terminal. kubectl apply
succeeds, the Deployment sits at 0/1, and the error is a FailedCreate event
on the ReplicaSet. Fix it by fixing the webhook’s pods; the endpoint check
above tells you whether any are ready. A pod that is Running but not Ready is
not an endpoint, for the reasons in
Pod is Running but 0/1 READY.
A related case: service "…" not found. Typically an operator was uninstalled
but its webhook configuration was not, and it still intercepts everything it used to.
If nothing should be serving that webhook any more, delete the configuration.
2. The API server cannot reach the pod
The ending is context deadline exceeded, the endpoints are fine, and the
webhook works from inside the cluster. The packets from the control plane are
being dropped. To show what that looks like, I pointed a Service at an address
nothing answers on (192.0.2.10, a documentation range):
Error from server (InternalError): error when creating "cm.yaml": Internal error occurred: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-blackhole.k32-webhook.svc:443/validate?timeout=10s": context deadline exceeded
kubectl apply -f cm.yaml 0.06s user 0.03s system 0% cpu 10.077 total
Word for word the same as a slow webhook. The difference is on the other end: a slow webhook received the request, and an unreachable one did not. If the webhook logs requests at all, its logs settle it.
The common versions:
- Managed control planes behind a firewall. On GKE clusters with network isolation (private clusters), the default firewall rules only let the control plane open connections to nodes and pods on ports 443 and 10250, and those are the pod’s ports, not the Service’s. A webhook Service on 443 that targets a container on 8443 or 9443 is blocked until you add a firewall rule for that port (GKE: Add firewall rules for specific use cases).
- A NetworkPolicy on the webhook’s namespace. A default-deny ingress policy can block the API server too, unless you allow it. It did not block it on my single-node kind cluster, because traffic from the node a pod runs on is always allowed; a pod in another namespace was blocked by the same policy. On a real cluster, where the API server is on another node or outside the cluster, allow its source addresses to the webhook’s container port.
- A self-managed control-plane node that cannot reach the pod network, for example because the CNI or kube-proxy is not running there. In-cluster requests work, and only the API server’s calls hang.
You can test the path from the API server without admission in the loop. The
API server’s service proxy picks a ready pod from the Service’s endpoints and
dials its IP directly (ResourceLocation in
pkg/registry/core/service/storage/storage.go, lines 452–487 on release-1.36):
kubectl get --raw '/api/v1/namespaces/k32-webhook/services/https:k32-hook:443/proxy/'
Error from server (InternalError): an error on the server ("<!DOCTYPE HTML>...<p>Error code: 501</p>...") has prevented the request from succeeding
That 501 page came from the webhook. Any HTTP answer, even an error, proves the
API server can reach the pod on that port; a hang points at the network. The
proxy does not verify certificates (InsecureSkipVerify in CreateProxyTransport,
pkg/controlplane/apiserver/config.go line 423 on release-1.36), so it tests
reachability, not your caBundle. It also always routes to the pod, so on a
ClusterIP-routing API server it skips kube-proxy.
3. The certificate does not match caBundle
The ending contains x509. Either caBundle is not the CA that signed the
webhook’s serving certificate, or it is empty, or the certificate was issued
for another name. Usual causes: cert-manager’s CA injector has not filled in
caBundle yet or is not running; the webhook rotated its certificate and
nothing updated the configuration; a chart was reinstalled and generated a new
CA. Compare what the configuration trusts with who signed the certificate the
webhook serves (here it is in a Secret, as it usually is):
kubectl get validatingwebhookconfiguration k32-policy -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -subject
kubectl -n k32-webhook get secret k32-hook-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -issuer
subject= /CN=some-other-ca
issuer= /CN=k32-webhook-ca
Different names: the bundle cannot verify that certificate. Put the right CA in
caBundle, or fix whatever is meant to inject it.
This x509 is the API server distrusting the webhook. It has nothing to do
with your kubeconfig, which is the subject of
x509: certificate signed by unknown authority, and kubectl’s other connection errors.
4. The webhook is slow
context deadline exceeded, and the webhook’s own logs show the request
arriving. It is waiting on something: an external policy service, an image
registry lookup, its own calls back to the API server, or too few replicas for
the load. Fix the webhook, and give it more replicas.
Raising timeoutSeconds towards 30 only moves the wait. Switching to
failurePolicy: Ignore does not remove it either. With the same backend
stuck and timeoutSeconds: 3:
configmap/app-config created
kubectl apply -f cm.yaml 0.07s user 0.03s system 3% cpu 3.105 total
The create succeeded, three seconds late, and the webhook was silently skipped. The only traces are in the API server’s log and metrics:
W0930 07:45:39.023819 1 dispatcher.go:205] Failed calling webhook, failing open k32-policy.k32.example.com: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=3s": context deadline exceeded
5. The webhook is blocking its own pods
The trap that turns a blip into an outage. A webhook with
failurePolicy: Fail whose rules cover pods, and whose namespaceSelector
matches its own namespace. While one webhook pod is running, nothing happens.
Once all of them are gone at the same time (a node restart, a scale to zero, a
cluster brought back up), the ReplicaSet asks to create a replacement, the API
server asks the webhook for permission, and there is no webhook to ask.
Reproduced with a webhook for pods scoped to its own namespace, scaled to zero and back to one:
kubectl -n k32-webhook get deploy k32-hook
kubectl -n k32-webhook get events --field-selector reason=FailedCreate -o custom-columns='OBJECT:.involvedObject.name,MESSAGE:.message'
NAME READY UP-TO-DATE AVAILABLE AGE
k32-hook 0/1 0 0 14m
OBJECT MESSAGE
k32-hook-66b5867657 Error creating: Internal error occurred: failed calling webhook "k32-pods.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=10s": dial tcp 10.96.38.100:443: connect: connection refused
It never recovers on its own. Look at what the webhook matches, and at the labels on its own namespace:
kubectl get validatingwebhookconfiguration k32-selflock -o jsonpath='{range .webhooks[*]}{.name}{"\t"}{.rules[*].resources}{"\t"}{.namespaceSelector}{"\t"}{.objectSelector}{"\n"}{end}'
kubectl get ns k32-webhook --show-labels
k32-pods.k32.example.com ["pods"] {"matchLabels":{"k32-demo":"webhook"}} {"matchLabels":{"app":"k32-hook"}}
NAME STATUS AGE LABELS
k32-webhook Active 14m k32-demo=webhook,kubernetes.io/metadata.name=k32-webhook
The lab webhook matches its own namespace on purpose. In the wild it is usually
by omission: an empty namespaceSelector, the default, matches every
namespace, including the webhook’s own. Fix it by excluding the webhook’s
namespace. The API server sets
kubernetes.io/metadata.name
on every namespace, so you do not need to add a label:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values: ["kube-system", "my-webhook-namespace"]
On the lab webhook I patched in the same NotIn, plus an Exists that keeps
it inside my labelled test namespaces on a shared cluster:
# selfix-patch.yaml
webhooks:
- name: k32-pods.k32.example.com
namespaceSelector:
matchLabels: null
matchExpressions:
- key: k32-demo
operator: Exists
- key: kubernetes.io/metadata.name
operator: NotIn
values: ["k32-webhook"]
kubectl patch validatingwebhookconfiguration k32-selflock --type=strategic --patch-file selfix-patch.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io/k32-selflock patched
Twenty-five seconds later:
kubectl -n k32-webhook get deploy,pods
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/k32-hook 1/1 1 1 15m
NAME READY STATUS RESTARTS AGE
pod/k32-hook-66b5867657-4cg6x 1/1 Running 0 9s
The ReplicaSet was still retrying, so the pod came back without anyone touching it. The same deadlock happens between two webhooks that check each other’s pods, or a webhook that intercepts the CNI or storage plugin it depends on; the Kubernetes docs call these dependency loops.
The emergency escape hatch
When nothing can be created and you cannot fix the webhook quickly, delete its
configuration. You always can: the API server never sends
ValidatingWebhookConfiguration or MutatingWebhookConfiguration objects to
webhooks (IsExemptAdmissionConfigurationResource,
predicates/rules/rules.go line 121, applied in generic/webhook.go line 358
on release-1.36 and 415 on release-1.37). Keep a copy first:
kubectl get validatingwebhookconfiguration k32-selflock -o yaml > k32-selflock-backup.yaml
kubectl delete validatingwebhookconfiguration k32-selflock
kubectl -n k32-webhook get deploy k32-hook
validatingwebhookconfiguration.admissionregistration.k8s.io "k32-selflock" deleted
NAME READY UP-TO-DATE AVAILABLE AGE
k32-hook 1/1 1 1 15m
The Deployment had been at 0/1; fifteen seconds after the delete it was
back. Once the webhook was healthy, kubectl apply -f k32-selflock-backup.yaml
put the configuration back as it was.
Know what you have turned off. Until it is restored, a validating webhook’s
policy is not enforced, and a mutating webhook’s changes are not made:
sidecars are not injected, defaults and labels are not added. Anything created
in the gap keeps that shape until it is recreated. Many operators also
reconcile their own webhook configurations and will put it back once they are
running, which is usually what you want. A gentler option is to patch
failurePolicy to Ignore for the duration, with the same consequences and a
timeout’s delay on every matching request.
Preventing it
- Scope every webhook.
namespaceSelectorexcluding its own namespace andkube-system,objectSelectorormatchConditionsfor everything else it does not need, and only the operations it needs. A pod mutator registered forUPDATEcan also stop pods from ever leavingTerminating, as covered in Pod stuck in Terminating. - Run at least two replicas with a PodDisruptionBudget, spread across nodes.
With a single replica and
failurePolicy: Fail, every node drain is a window in which matching requests fail. - Choose
failurePolicyper webhook.Failfor anything that enforces security,Ignorefor conveniences. KeeptimeoutSecondssmall. - Alert on the metrics if you can read the API server’s. They count failures to call by webhook name, and every time a webhook was skipped:
kubectl get --raw /metrics | grep -E '^apiserver_admission_webhook_(rejection_count|fail_open_count)' | grep k32
apiserver_admission_webhook_fail_open_count{name="k32-policy.k32.example.com",type="validating"} 1
apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error",name="k32-pods.k32.example.com",operation="CREATE",rejection_code="503",type="validating"} 25
apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error",name="k32-policy.k32.example.com",operation="CREATE",rejection_code="503",type="validating"} 12
The metric records 503, the status attached to the transport error
(dispatcher.go line 309); the client still receives the 500 shown above.
When a Deployment sits at 0/1 because of a webhook, the reason is a FailedCreate warning on its ReplicaSet. KubeGlance is a native Kubernetes client for iPhone and iPad, with a full Mac app on the same core, and its cluster-wide warnings feed shows that event without knowing which ReplicaSet to ask.
Get KubeGlanceThe Kubernetes dashboard that fits in your pocket
KubeGlance is a native Kubernetes client for iPhone and iPad — the real dashboard, not a companion — with a full Mac app on the same core. Pods, workloads, logs and events, straight from your kubeconfig.
Download KubeGlance

