KubeGlanceDownload
Internal error occurred: failed calling webhook: context deadline exceeded, and every other ending

troubleshooting

Internal error occurred: failed calling webhook: context deadline exceeded, and every other ending

failed calling webhook ... context deadline exceeded: the API server could not get an answer from an admission webhook. What each ending means and the fix.

· 11 min read

Your object was never judged. Before storing it, the API server has to send it to an admission webhook, and that call failed: the API server could not reach the webhook, could not trust it, or did not get an answer within timeoutSeconds. Because the webhook’s failurePolicy is Fail, which is the default, the API server refuses the whole request and reports it as its own failure: HTTP 500, Internal error occurred.

The part that tells you what went wrong is the text after the URL. For context deadline exceeded, it means the API server waited the full timeout and got nothing back:

time kubectl apply -f cm.yaml
Error from server (InternalError): error when creating "cm.yaml": Internal error occurred: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=10s": context deadline exceeded
kubectl apply -f cm.yaml  0.07s user 0.03s system 0% cpu 10.088 total

Ten seconds exactly: the default timeout, spent before the ConfigMap was even looked at. Every output here is from a kind cluster on Kubernetes 1.36.1, with a small TLS webhook of my own in front of ConfigMaps. The code paths are source-checked on release-1.36 and release-1.37.

The fastest check

1. Read the ending. The same code path produces all of these. Each was reproduced on 1.36.1 except the second, which is quoted from source:

Text after the URLWhat happened
context deadline exceededNo reply within timeoutSeconds. The webhook is slow, or the packets never arrive
no endpoints available for service "k32-hook"No ready webhook pod, on an API server that routes to endpoints
dial tcp 10.96.38.100:443: connect: connection refusedNo ready webhook pod, on an API server that routes to the ClusterIP (the default)
service "k32-hook-typo" not foundThe configuration names a Service that does not exist
no service port 8443 found for service "k32-hook"clientConfig.service.port is not a port on the Service. It wants the Service port, not the container port
tls: failed to verify certificate: x509: certificate signed by unknown authoritycaBundle is wrong or empty
tls: failed to verify certificate: x509: certificate is valid for k32-hook.default.svc, not k32-hook.k32-webhook.svcThe serving certificate was issued for another name

One more comes from the same place with a slightly different shape, because the webhook did answer: failed calling webhook "…": received invalid webhook response: expected response.uid="a50c77b4-…", got "not-the-request-uid". And one that is not this problem at all: admission webhook "…" denied the request: … means the call worked and the webhook said no. Read its reason and fix your object.

2. Find the configuration. The name in quotes is a webhook name, not the name of the configuration that contains it. List both, with where they send traffic and what happens when that fails:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o custom-columns='CONFIG:.metadata.name,WEBHOOK:.webhooks[*].name,SERVICE:.webhooks[*].clientConfig.service.name,NAMESPACE:.webhooks[*].clientConfig.service.namespace,POLICY:.webhooks[*].failurePolicy,TIMEOUT:.webhooks[*].timeoutSeconds'
CONFIG         WEBHOOK                        SERVICE    NAMESPACE     POLICY   TIMEOUT
k32-selflock   k32-pods.k32.example.com       k32-hook   k32-webhook   Fail     10
k32-defaults   k32-defaults.k32.example.com   k32-hook   k32-webhook   Ignore   5

3. Check the Service has somewhere to send it:

kubectl -n k32-webhook get endpointslices -l kubernetes.io/service-name=k32-hook
NAME             ADDRESSTYPE   PORTS     ENDPOINTS   AGE
k32-hook-wnks8   IPv4          <unset>   <unset>     5m44s

<unset> means no ready pod. Go to the webhook’s Deployment. If there are endpoints and you still get context deadline exceeded, the problem is between the API server and the pod, or inside the webhook.

What the API server is doing

The webhook is called by kube-apiserver itself, from wherever kube-apiserver runs. Not by kubectl, and not from inside a pod. That one fact explains most of this error.

Every ending in the table is produced at one step of this call. Where it stops tells you which component to look at.

Where it sits in the request. A create runs mutating admission first (endpoints/handlers/create.go line 203), then the resource’s own validation (rest.BeforeCreate, registry/generic/registry/store.go line 509 on release-1.36), then validating admission (line 515). Mutating webhooks are called one after another in a loop (mutating/dispatcher.go line 127); validating webhooks are called in parallel, one goroutine each (validating/dispatcher.go line 130). Slow mutating webhooks add their waits together; slow validating webhooks overlap.

How it finds the pod. The URL in the message, https://k32-hook.k32-webhook.svc:443, is built from clientConfig.service (pkg/util/webhook/client.go line 187 on release-1.36), but the API server never looks that name up in DNS. Its dialer swaps in an address from a service resolver (line 204; on 1.37 a wrapping round tripper does the same job by default). Which resolver depends on one flag, in buildServiceResolver (cmd/kube-apiserver/app/server.go lines 283–300):

  • Default: the Service’s ClusterIP. The connection then goes through kube-proxy on the API server’s own node, which is why an empty Service shows up as connection refused: kube-proxy installs a REJECT rule for a Service with no endpoints (pkg/proxy/iptables/proxier.go line 910; the nftables proxier does the same).
  • --enable-aggregator-routing=true: a ready endpoint picked from the EndpointSlices, dialled directly. No ready endpoint gives no endpoints available for service (pkg/util/proxy/proxy.go lines 82 and 126 on release-1.36, 78 and 149 on release-1.37). k3s turns this flag on unless its egress selector is disabled (pkg/daemons/control/server.go line 239, default mode agent in pkg/cli/cmds/server.go), and the string turns up verbatim in reports from Rancher (rancher/rancher#54739), EKS (aws-load-balancer-controller#2336) and OpenShift (Red Hat solution 7051156). Same cause as connection refused, different wording. In 1.37 this path also load-balances across endpoints instead of reusing one connection (#139237, feature gate WebhookRoundTripLoadBalancing, beta and on by default).

What it trusts. TLS is verified against caBundle only, with the server name forced to <service>.<namespace>.svc (client.go lines 155 and 194 on release-1.36). The certificate must be signed by that bundle and valid for that exact name.

How long it waits. timeoutSeconds defaults to 10 (pkg/apis/admissionregistration/v1/defaults.go lines 50 and 74) and must be between 1 and 30 (validation.go line 388):

The ValidatingWebhookConfiguration "k32-policy" is invalid: webhooks[0].timeoutSeconds: Invalid value: 31: the timeout value must be between 1 and 30 seconds

The dispatcher puts that timeout on the call’s context (dispatcher.go line 279) and also passes it to the webhook as the ?timeout=10s in the URL, so a well-behaved webhook knows its budget. Setting timeoutSeconds: 3 against the same stuck backend:

Error from server (InternalError): error when creating "cm.yaml": Internal error occurred: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=3s": context deadline exceeded
kubectl apply -f cm.yaml  0.07s user 0.03s system 3% cpu 3.089 total

Why it says “Internal error”. The transport failure becomes failed calling webhook "<name>": failed to call webhook: <Go error> (pkg/util/webhook/error.go line 36, validating/dispatcher.go line 311). With failurePolicy: Fail, the default (defaults.go lines 33 and 57), it is wrapped in NewInternalError (validating/dispatcher.go line 218, mutating/dispatcher.go line 226), which prefixes Internal error occurred: and sets status 500 (apimachinery/pkg/api/errors/errors.go line 395). The server is telling you that it failed, not that your object is bad. The line numbers are the same on release-1.37 unless noted. The failed to call webhook: middle part was added in 1.23; older API servers print the Go error straight after the webhook name.

Why it happens, and the fix for each

1. No ready webhook pods

Rule this out first. The webhook’s Deployment is scaled down, crash-looping, not yet ready after an install or upgrade, or its only node is gone. The ending is connection refused or no endpoints available for service, and it fails instantly rather than after ten seconds. Scaled to zero:

Error from server (InternalError): error when creating "cm.yaml": Internal error occurred: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=10s": dial tcp 10.96.38.100:443: connect: connection refused
kubectl apply -f cm.yaml  0.05s user 0.02s system 106% cpu 0.064 total

Behind a Deployment you will not see this at the terminal. kubectl apply succeeds, the Deployment sits at 0/1, and the error is a FailedCreate event on the ReplicaSet. Fix it by fixing the webhook’s pods; the endpoint check above tells you whether any are ready. A pod that is Running but not Ready is not an endpoint, for the reasons in Pod is Running but 0/1 READY.

A related case: service "…" not found. Typically an operator was uninstalled but its webhook configuration was not, and it still intercepts everything it used to. If nothing should be serving that webhook any more, delete the configuration.

2. The API server cannot reach the pod

The ending is context deadline exceeded, the endpoints are fine, and the webhook works from inside the cluster. The packets from the control plane are being dropped. To show what that looks like, I pointed a Service at an address nothing answers on (192.0.2.10, a documentation range):

Error from server (InternalError): error when creating "cm.yaml": Internal error occurred: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-blackhole.k32-webhook.svc:443/validate?timeout=10s": context deadline exceeded
kubectl apply -f cm.yaml  0.06s user 0.03s system 0% cpu 10.077 total

Word for word the same as a slow webhook. The difference is on the other end: a slow webhook received the request, and an unreachable one did not. If the webhook logs requests at all, its logs settle it.

The common versions:

  • Managed control planes behind a firewall. On GKE clusters with network isolation (private clusters), the default firewall rules only let the control plane open connections to nodes and pods on ports 443 and 10250, and those are the pod’s ports, not the Service’s. A webhook Service on 443 that targets a container on 8443 or 9443 is blocked until you add a firewall rule for that port (GKE: Add firewall rules for specific use cases).
  • A NetworkPolicy on the webhook’s namespace. A default-deny ingress policy can block the API server too, unless you allow it. It did not block it on my single-node kind cluster, because traffic from the node a pod runs on is always allowed; a pod in another namespace was blocked by the same policy. On a real cluster, where the API server is on another node or outside the cluster, allow its source addresses to the webhook’s container port.
  • A self-managed control-plane node that cannot reach the pod network, for example because the CNI or kube-proxy is not running there. In-cluster requests work, and only the API server’s calls hang.

You can test the path from the API server without admission in the loop. The API server’s service proxy picks a ready pod from the Service’s endpoints and dials its IP directly (ResourceLocation in pkg/registry/core/service/storage/storage.go, lines 452–487 on release-1.36):

kubectl get --raw '/api/v1/namespaces/k32-webhook/services/https:k32-hook:443/proxy/'
Error from server (InternalError): an error on the server ("<!DOCTYPE HTML>...<p>Error code: 501</p>...") has prevented the request from succeeding

That 501 page came from the webhook. Any HTTP answer, even an error, proves the API server can reach the pod on that port; a hang points at the network. The proxy does not verify certificates (InsecureSkipVerify in CreateProxyTransport, pkg/controlplane/apiserver/config.go line 423 on release-1.36), so it tests reachability, not your caBundle. It also always routes to the pod, so on a ClusterIP-routing API server it skips kube-proxy.

3. The certificate does not match caBundle

The ending contains x509. Either caBundle is not the CA that signed the webhook’s serving certificate, or it is empty, or the certificate was issued for another name. Usual causes: cert-manager’s CA injector has not filled in caBundle yet or is not running; the webhook rotated its certificate and nothing updated the configuration; a chart was reinstalled and generated a new CA. Compare what the configuration trusts with who signed the certificate the webhook serves (here it is in a Secret, as it usually is):

kubectl get validatingwebhookconfiguration k32-policy -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -subject
kubectl -n k32-webhook get secret k32-hook-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -issuer
subject= /CN=some-other-ca
issuer= /CN=k32-webhook-ca

Different names: the bundle cannot verify that certificate. Put the right CA in caBundle, or fix whatever is meant to inject it.

This x509 is the API server distrusting the webhook. It has nothing to do with your kubeconfig, which is the subject of x509: certificate signed by unknown authority, and kubectl’s other connection errors.

4. The webhook is slow

context deadline exceeded, and the webhook’s own logs show the request arriving. It is waiting on something: an external policy service, an image registry lookup, its own calls back to the API server, or too few replicas for the load. Fix the webhook, and give it more replicas.

Raising timeoutSeconds towards 30 only moves the wait. Switching to failurePolicy: Ignore does not remove it either. With the same backend stuck and timeoutSeconds: 3:

configmap/app-config created
kubectl apply -f cm.yaml  0.07s user 0.03s system 3% cpu 3.105 total

The create succeeded, three seconds late, and the webhook was silently skipped. The only traces are in the API server’s log and metrics:

W0930 07:45:39.023819       1 dispatcher.go:205] Failed calling webhook, failing open k32-policy.k32.example.com: failed calling webhook "k32-policy.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=3s": context deadline exceeded

5. The webhook is blocking its own pods

The trap that turns a blip into an outage. A webhook with failurePolicy: Fail whose rules cover pods, and whose namespaceSelector matches its own namespace. While one webhook pod is running, nothing happens. Once all of them are gone at the same time (a node restart, a scale to zero, a cluster brought back up), the ReplicaSet asks to create a replacement, the API server asks the webhook for permission, and there is no webhook to ask.

Reproduced with a webhook for pods scoped to its own namespace, scaled to zero and back to one:

kubectl -n k32-webhook get deploy k32-hook
kubectl -n k32-webhook get events --field-selector reason=FailedCreate -o custom-columns='OBJECT:.involvedObject.name,MESSAGE:.message'
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
k32-hook   0/1     0            0           14m
OBJECT                MESSAGE
k32-hook-66b5867657   Error creating: Internal error occurred: failed calling webhook "k32-pods.k32.example.com": failed to call webhook: Post "https://k32-hook.k32-webhook.svc:443/validate?timeout=10s": dial tcp 10.96.38.100:443: connect: connection refused

It never recovers on its own. Look at what the webhook matches, and at the labels on its own namespace:

kubectl get validatingwebhookconfiguration k32-selflock -o jsonpath='{range .webhooks[*]}{.name}{"\t"}{.rules[*].resources}{"\t"}{.namespaceSelector}{"\t"}{.objectSelector}{"\n"}{end}'
kubectl get ns k32-webhook --show-labels
k32-pods.k32.example.com	["pods"]	{"matchLabels":{"k32-demo":"webhook"}}	{"matchLabels":{"app":"k32-hook"}}
NAME          STATUS   AGE   LABELS
k32-webhook   Active   14m   k32-demo=webhook,kubernetes.io/metadata.name=k32-webhook

The lab webhook matches its own namespace on purpose. In the wild it is usually by omission: an empty namespaceSelector, the default, matches every namespace, including the webhook’s own. Fix it by excluding the webhook’s namespace. The API server sets kubernetes.io/metadata.name on every namespace, so you do not need to add a label:

namespaceSelector:
  matchExpressions:
  - key: kubernetes.io/metadata.name
    operator: NotIn
    values: ["kube-system", "my-webhook-namespace"]

On the lab webhook I patched in the same NotIn, plus an Exists that keeps it inside my labelled test namespaces on a shared cluster:

# selfix-patch.yaml
webhooks:
- name: k32-pods.k32.example.com
  namespaceSelector:
    matchLabels: null
    matchExpressions:
    - key: k32-demo
      operator: Exists
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values: ["k32-webhook"]
kubectl patch validatingwebhookconfiguration k32-selflock --type=strategic --patch-file selfix-patch.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io/k32-selflock patched

Twenty-five seconds later:

kubectl -n k32-webhook get deploy,pods
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/k32-hook   1/1     1            1           15m

NAME                            READY   STATUS    RESTARTS   AGE
pod/k32-hook-66b5867657-4cg6x   1/1     Running   0          9s

The ReplicaSet was still retrying, so the pod came back without anyone touching it. The same deadlock happens between two webhooks that check each other’s pods, or a webhook that intercepts the CNI or storage plugin it depends on; the Kubernetes docs call these dependency loops.

The emergency escape hatch

When nothing can be created and you cannot fix the webhook quickly, delete its configuration. You always can: the API server never sends ValidatingWebhookConfiguration or MutatingWebhookConfiguration objects to webhooks (IsExemptAdmissionConfigurationResource, predicates/rules/rules.go line 121, applied in generic/webhook.go line 358 on release-1.36 and 415 on release-1.37). Keep a copy first:

kubectl get validatingwebhookconfiguration k32-selflock -o yaml > k32-selflock-backup.yaml
kubectl delete validatingwebhookconfiguration k32-selflock
kubectl -n k32-webhook get deploy k32-hook
validatingwebhookconfiguration.admissionregistration.k8s.io "k32-selflock" deleted
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
k32-hook   1/1     1            1           15m

The Deployment had been at 0/1; fifteen seconds after the delete it was back. Once the webhook was healthy, kubectl apply -f k32-selflock-backup.yaml put the configuration back as it was.

Know what you have turned off. Until it is restored, a validating webhook’s policy is not enforced, and a mutating webhook’s changes are not made: sidecars are not injected, defaults and labels are not added. Anything created in the gap keeps that shape until it is recreated. Many operators also reconcile their own webhook configurations and will put it back once they are running, which is usually what you want. A gentler option is to patch failurePolicy to Ignore for the duration, with the same consequences and a timeout’s delay on every matching request.

Preventing it

Which ending you have decides where to look next.
  • Scope every webhook. namespaceSelector excluding its own namespace and kube-system, objectSelector or matchConditions for everything else it does not need, and only the operations it needs. A pod mutator registered for UPDATE can also stop pods from ever leaving Terminating, as covered in Pod stuck in Terminating.
  • Run at least two replicas with a PodDisruptionBudget, spread across nodes. With a single replica and failurePolicy: Fail, every node drain is a window in which matching requests fail.
  • Choose failurePolicy per webhook. Fail for anything that enforces security, Ignore for conveniences. Keep timeoutSeconds small.
  • Alert on the metrics if you can read the API server’s. They count failures to call by webhook name, and every time a webhook was skipped:
kubectl get --raw /metrics | grep -E '^apiserver_admission_webhook_(rejection_count|fail_open_count)' | grep k32
apiserver_admission_webhook_fail_open_count{name="k32-policy.k32.example.com",type="validating"} 1
apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error",name="k32-pods.k32.example.com",operation="CREATE",rejection_code="503",type="validating"} 25
apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error",name="k32-policy.k32.example.com",operation="CREATE",rejection_code="503",type="validating"} 12

The metric records 503, the status attached to the transport error (dispatcher.go line 309); the client still receives the 500 shown above.

When a Deployment sits at 0/1 because of a webhook, the reason is a FailedCreate warning on its ReplicaSet. KubeGlance is a native Kubernetes client for iPhone and iPad, with a full Mac app on the same core, and its cluster-wide warnings feed shows that event without knowing which ReplicaSet to ask.

Get KubeGlance
#admission-webhooks #kube-apiserver #failurepolicy #tls #troubleshooting

The Kubernetes dashboard that fits in your pocket

KubeGlance is a native Kubernetes client for iPhone and iPad — the real dashboard, not a companion — with a full Mac app on the same core. Pods, workloads, logs and events, straight from your kubeconfig.

Download KubeGlance