x509: certificate signed by unknown authority, and kubectl's other connection errors

Connecting kubectl to a cluster is four things in sequence: reach the address, agree on TLS, prove who you are, and be allowed to do the thing. Each one fails with its own message, and the messages look far more alike than the problems do.

Naming the layer takes ten seconds and eliminates most of the search space. Doing it in the wrong order is how people end up adding insecure-skip-tls-verify: true to fix an expired token.

Zoom

What actually happens on the wire

The four layers are not a mental model someone invented for a blog post. They are the order of operations in a single request, and each error is emitted by the first step that fails — which is why the later ones only appear once the earlier ones succeed.

Zoom

Layer 1: Unable to connect to the server: dial tcp

Unable to connect to the server: dial tcp 10.0.1.4:6443: i/o timeout

Nothing about Kubernetes has happened yet. The TCP connection did not open, and the address in the message is the one kubectl actually used — which is the first thing to check, because it is frequently not the cluster you had in mind.

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

--minify restricts the output to the current context, which matters when your KUBECONFIG merges several files and the current context is not the one you believe it is. That merge has its own set of surprises, which we went through in how KUBECONFIG merges multiple files.

Then, in order of how often it turns out to be the answer:

• The VPN is down, or the API server is on a private endpoint you are not inside. Endpoints named *.internal, 10.* or 172.16-31.* are the tell.
• The wrong context. A dead staging cluster from six months ago is still in your kubeconfig and still the current context.
• The cluster moved. Recreated clusters get new endpoints; the old kubeconfig does not update itself.
• Cloud firewall rules. Managed control planes commonly have an authorised-networks list, and your home IP changed.

i/o timeout means packets went nowhere — a firewall dropping silently, or a route that does not exist. connection refused is the opposite and more useful: something answered, so you have the right host and the wrong port, or the API server is not running.

Layer 2: x509: certificate signed by unknown authority

Unable to connect to the server: x509: certificate signed by unknown authority

TCP worked. TLS did not. kubectl was offered a certificate it cannot chain back to a CA it trusts.

This one is worth understanding rather than working around, because the standard workaround disables the only protection you have.

Confirm it at the TLS layer

openssl s_client -connect <host>:<port> </dev/null

The port comes from the server: URL — 6443 on most self-managed clusters, 443 on several managed ones.

Against a cluster whose CA your machine does not have, that prints:

depth=0 CN = kube-apiserver
verify error:num=20:unable to get local issuer certificate
verify return:1
subject=/CN=kube-apiserver
issuer=/CN=kubernetes

That is the same failure kubectl reports, one layer down and with more detail. The issuer line names the CA that signed the API server’s certificate — a cluster-internal CA, not a public one. Your system trust store has never heard of it and never will. The CA has to come from your kubeconfig.

The four causes

1. certificate-authority-data is missing from the cluster entry. Check:

kubectl config view --minify | grep -E 'certificate-authority|server:'

A present CA appears as certificate-authority-data: DATA+OMITTED — kubectl redacts the value unless you pass --raw. If neither certificate-authority-data nor certificate-authority is there at all, kubectl falls back to your system trust store, which cannot possibly contain a cluster CA. This is what usually happens when a kubeconfig is assembled by hand or reduced by a script.

2. The cluster’s CA was rotated. Your kubeconfig has the old one. Nothing you do locally fixes this; you need a fresh kubeconfig from whoever runs the cluster.

3. Something is intercepting TLS. Corporate proxies, some VPN clients and endpoint-security agents re-sign traffic with their own CA. The issuer line from openssl s_client names them directly, and it will not say kube-apiserver. The fix is to add the corporate CA to your trust store, not to disable verification.

4. Wrong context again. You are presenting cluster A’s CA to cluster B.

Why --insecure-skip-tls-verify is not a fix

It silences the error by no longer checking who you are talking to. Your bearer token or client certificate is then sent to whatever answered on that address. On a public endpoint that is a credential-disclosure bug, not a workaround. There is one legitimate use — reading the CA off a cluster you are bootstrapping — and it should never survive into a saved kubeconfig.

If you have a copy of the correct CA, point at it explicitly instead:

kubectl config set-cluster <cluster> --certificate-authority=/path/to/ca.crt --embed-certs=true

Layer 2b: x509: certificate has expired or is not yet valid

Same layer, different problem, and the message tells you which end:

Unable to connect to the server: x509: certificate has expired or is not yet
valid: current time 2026-08-28T10:14:22+01:00 is after 2026-08-27T09:11:03Z

Read both timestamps. If the “current time” is wrong, your clock is wrong — common on a laptop that has been suspended for a week, and on a VM restored from a snapshot. Fix the clock and the error goes away.

If the current time is right, something genuinely expired. To check your client certificate:

kubectl config view --minify --raw -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d | openssl x509 -noout -subject -dates

subject= /O=system:masters/CN=kubernetes-admin
notBefore=Aug 27 09:11:03 2025 GMT
notAfter=Aug 27 09:11:03 2026 GMT

kubeadm issues client certificates with a one-year lifetime by default, and renews them on control-plane upgrade. A cluster that has not been upgraded in a year hands out this error to everyone at once, which at least makes it easy to recognise.

The subject line is also the answer to “who does this kubeconfig think I am” — CN is the username and O supplies the groups, which is what RBAC binds against. If that is not what you expected, the error you are about to hit next is a Forbidden, not a TLS one.

Layer 3: You must be logged in to the server (Unauthorized)

error: You must be logged in to the server (Unauthorized)

TLS succeeded. The API server received your request, looked at your credentials and could not tell who you are. This is HTTP 401 — authentication, not permissions.

kubectl auth whoami

ATTRIBUTE   VALUE
Username    kubernetes-admin
Groups      [system:masters system:authenticated]

That has been stable since 1.28 and is the fastest way to find out what the server thinks of your credentials. If it fails with the same Unauthorized, nothing in your kubeconfig is being accepted.

Causes, in order:

• An expired token. Since 1.24 the API server no longer auto-creates a never-expiring Secret for a ServiceAccount; projected tokens are time-bound and audience-bound. A token pasted into a kubeconfig months ago is dead, and it fails with exactly this message rather than with anything mentioning expiry.
• An exec plugin that is failing. aws eks get-token, gke-gcloud-auth-plugin, kubelogin and friends run a binary to mint credentials. If that binary is missing, unauthenticated, or not on the PATH kubectl inherited, you get the same 401. Run the command in users[].user.exec by hand and read its error — it is almost always clearer than kubectl’s.
• A user entry that does not match the context. Contexts pair a cluster with a user, and a hand-edited kubeconfig can easily pair them wrongly.
• A rotated or deleted service account.

To see the request and response rather than guessing:

kubectl get ns --v=6

Level 6 logs the URL and status code of every request. Level 8 adds headers and bodies, which is more than you usually need and includes your token, so keep it out of anything you paste into a ticket.

Layer 4: Error from server (Forbidden)

Error from server (Forbidden): pods is forbidden: User "dev@example.com" cannot
list resource "pods" in API group "" in the namespace "prod"

This is not a connection problem at all. You are authenticated; you are just not allowed. It is HTTP 403, and the message names the user, the verb, the resource and the namespace — everything you need to write the rule that would permit it.

kubectl auth can-i list pods -n prod
kubectl auth can-i --list -n prod

The second prints everything you can do in that namespace, which is more useful than checking one verb at a time. What to grant, and the traps in the built-in view role, are covered in RBAC for a read-only Kubernetes user.

A checklist that fits in your head

Work upward and stop at the first layer that fails. Everything above a broken layer reports failure too, which is why fixing the top one never works.