KubeGlanceDownload
DNS i/o timeout after a default-deny NetworkPolicy: the egress rule you need

troubleshooting

DNS i/o timeout after a default-deny NetworkPolicy: the egress rule you need

"lookup … on 10.96.0.10:53: i/o timeout" after a default-deny NetworkPolicy: why DNS breaks while pods stay Ready, and the rule that fixes it.

· 8 min read

If every name lookup in a namespace started timing out right after someone applied a NetworkPolicy, the policy cut off DNS. A pod that a policy isolates for egress may only send traffic the policies list, and a default-deny lists nothing, so lookups to CoreDNS count too. Every query is dropped, the resolver waits out its timeouts, and the application reports a name it cannot resolve. Pods carry on Running and Ready the whole time.

The fix is one more policy: allow egress to the CoreDNS pods, on port 53, over UDP and TCP. Two versions of it look right and do not work. Both are below.

The fastest check

Run a lookup from inside an affected pod:

kubectl exec -n shop deploy/checkout -- nslookup payments

On Kubernetes 1.36.1 with Calico 3.31.4 (Canal), after a default-deny egress policy:

;; communications error to 10.96.0.10#53: timed out
;; communications error to 10.96.0.10#53: timed out
;; communications error to 10.96.0.10#53: timed out
;; no servers could be reached

The same failure, in the words of whatever is asking:

ClientWhat it printsTime to fail
Go program (net resolver)lookup payments on 10.96.0.10:53: read udp 10.244.1.7:38923->10.96.0.10:53: i/o timeout40.0 s
curlcurl: (6) Could not resolve host: payments11.3 s
getent hosts payments (glibc)nothing; exit status 210.0 s
dig / nslookup (BIND 9.18);; communications error to 10.96.0.10#53: timed out15.0 s
busybox nslookup (in an ephemeral container);; connection timed out; no servers could be reached

A Go service usually prints that line with its own prefix, such as dial tcp: lookup payments on 10.96.0.10:53: …. The part to recognise is a read that timed out against your cluster DNS address on port 53.

Then compare what you see with what the cluster says:

kubectl get pods -n shop
kubectl get networkpolicy -n shop
NAME                        READY   STATUS    RESTARTS   AGE
checkout-67779ccf47-8m59d   1/1     Running   0          8m55s
payments-6b46c5746-2j4v6    1/1     Running   0          8m55s

NAME                  POD-SELECTOR   AGE
default-deny-egress   <none>         8m43s

Everything is green, and there is a policy with an empty pod selector. That combination is the answer. A pod that stays Ready while its application cannot reach anything is a family of problems of its own, covered in pod Running but not Ready. Here the probe passes because the kubelet’s probe comes in to the pod, and the policy only restricts what goes out.

If the image has no shell or no lookup tools, an ephemeral container shares the pod’s network namespace, so the same policy applies to it:

kubectl debug -n shop checkout-67779ccf47-8m59d --profile=general --image=busybox:1.37 \
  -c dnscheck -- nslookup payments.shop.svc.cluster.local
kubectl logs -n shop checkout-67779ccf47-8m59d -c dnscheck

kubectl debug needs a pod name. It rejects deploy/checkout with "apps/v1, Kind=Deployment" not supported by debug.

Why it happens

The Kubernetes documentation says it in a caution box under Default deny all egress traffic: “A default deny-all egress policy also blocks DNS traffic.” The policy that causes it is the one most guides start with:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
  - Egress

podSelector: {} selects every pod in the namespace. policyTypes: [Egress] with no egress rules isolates them for egress and allows nothing. NetworkPolicies are additive allow lists: once a pod is isolated, a connection passes only if some policy that selects the pod allows it. A DNS query is a connection like any other.

What makes it confusing is how the failure looks from inside the pod:

The packet is rewritten to a CoreDNS pod address before the policy sees it, then dropped without a reply. The resolver only learns by waiting.

Three details in that sequence explain everything you see:

  • The drop is silent. No ICMP error and no TCP reset come back, so the resolver cannot tell a blocked query from a slow server. It waits the whole timeout and retries. glibc’s defaults are a 5-second timeout and 2 attempts (resolv.conf(5)), which is the 10 seconds getent took above. Go’s resolver does not stop at the first timeout: it moves on through the ndots:5 search list, and took 40 seconds to give up on one short name. Each request that needs a fresh lookup pays those seconds before it fails, so the first symptom is often latency or upstream timeouts rather than a clear error.
  • The pod looks healthy. Nothing in the pod’s status reflects which policies select it. If the application exits when a lookup fails at startup, you get CrashLoopBackOff instead, and the logs say “could not resolve”, which reads like a DNS outage rather than a policy.
  • The destination has already been rewritten. The pod sends to the kube-dns Service IP (10.96.0.10 here), and kube-proxy translates that to one CoreDNS pod’s IP before the packet leaves the node. On Canal the policy is checked after that rewrite. That is why the first “obvious” fix below does nothing.

The fix: allow egress to the CoreDNS pods, UDP and TCP

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Apply it next to the default-deny, not instead of it. Policies add up, so the deny still blocks everything else. On the same cluster, a minute later:

$ dig +short payments.shop.svc.cluster.local
10.96.217.27
$ dig +tcp +short payments.shop.svc.cluster.local
10.96.217.27

What each part does:

  • kubernetes.io/metadata.name: kube-system selects the namespace by name. The control plane sets this label on every namespace, and you cannot remove or change it (Targeting a Namespace by its name). Any other namespace label is one somebody has to remember to add.
  • namespaceSelector and podSelector in the same list item means pods labelled k8s-app=kube-dns in kube-system. Written as two items (a - in front of each), they would mean anything in kube-system, or any pod labelled k8s-app=kube-dns in this namespace, which is a different and much wider rule.
  • k8s-app: kube-dns is the label kubeadm, kind and most distributions give the CoreDNS pods. The Service is named kube-dns for historical reasons. Check yours before relying on it:
    kubectl get svc -n kube-system kube-dns -o jsonpath='{.spec.selector}'
    
    Whatever that prints is what the rule’s podSelector must say.

Once DNS works, the error changes. The same curl now fails with curl: (28) Connection timed out after 10003 milliseconds. That one is the default-deny doing its job on the actual connection to payments, which needs its own allow rule.

The fixes that look right and do not work

Allowing the Service IP with ipBlock

  egress:
  - to:
    - ipBlock:
        cidr: 10.96.0.10/32

On our Canal cluster this left every lookup timing out, exactly as before. The policy sees the CoreDNS pod’s address, not 10.96.0.10. The Kubernetes docs do not promise either behaviour: whether address rewriting “happens before or after NetworkPolicy processing” is not defined, and “may be different for different combinations of network plugin, cloud provider, Service implementation” (Network Policies). A rule that selects the DNS pods works on any plugin. A rule for the Service IP works on some.

UDP only

The same rule with only the UDP port:

$ dig +short payments.shop.svc.cluster.local
10.96.217.27
$ dig +tcp +short payments.shop.svc.cluster.local
;; Connection to 10.96.0.10#53(10.96.0.10) for payments.shop.svc.cluster.local failed: timed out.
;; no servers could be reached

Most lookups go over UDP, so this passes a quick test. A response too large for a UDP packet, such as a long list of addresses for a headless Service, comes back truncated, and the resolver then asks again over TCP. That TCP query is blocked, so the failure only appears for some names, some of the time. RFC 7766 is blunt about it: “All general-purpose DNS implementations MUST support both UDP and TCP transport.” Allow both.

Forgetting the other end

A policy in your own namespace is not the only one that can cut DNS. A default-deny for ingress in kube-system isolates the CoreDNS pods, and breaks DNS for every namespace in the cluster, including ones with no policies at all. From a namespace without a single policy, after default-deny-ingress was applied in kube-system:

$ dig +short +time=2 +tries=1 web.np-b.svc.cluster.local
;; communications error to 10.96.0.10#53: timed out
;; no servers could be reached

If you lock down kube-system, the CoreDNS pods need an ingress rule allowing UDP and TCP 53 from every namespace (namespaceSelector: {}).

Finding it before it bites

There is no kubectl view of “which pods are egress-isolated without DNS”, but the policies themselves hold enough to narrow it down. This lists every policy that restricts egress, with the DNS protocols it allows:

kubectl get networkpolicy -A -o json | jq -r '.items[]
  | select(any(.spec.policyTypes[]?; . == "Egress"))
  | [.metadata.namespace, .metadata.name,
     ([.spec.egress[]?.ports[]? | select(.port == 53 or .port == "dns" or .port == "dns-tcp") | .protocol // "TCP"]
      | unique | join(",") | if . == "" then "-" else . end)]
  | @tsv' | column -t
shop  allow-dns            TCP,UDP
shop  default-deny-egress  -

A namespace whose lines all end in - restricts egress and allows DNS nowhere. That is a strong hint, not a proof. Policies select pods by label, so an allow rule can exist and still miss the pod that needs it, and a rule with no ports allows every port to its peers. The proof is a lookup from the pod itself, as in the first section. Run it after every policy change, not only when something breaks.

KubeGlance policy coverage on iPhone: denied DNS traffic to a CoreDNS pod at the top, then a safety finding saying a StatefulSet cannot resolve names because its egress policy has no rule for kube-dns on 53/UDP, with buttons to copy a suggested policy and dry-run the fix
The DNS check in KubeGlance, on demo data: the policy that isolates the workload, the rule that is missing, and a suggested fix to dry-run.

KubeGlance runs this check for you. Its network policy coverage view works out which pods every policy selects, and flags any workload that egress isolation has cut off from the cluster’s DNS pods, naming the policy responsible. It suggests a fix, and lets you dry-run it against the cluster’s current pods before you apply it. Where Calico Whisker runs, the denied lookups show up there as well.

A default-deny that cut off DNS leaves every pod Running and Ready. KubeGlance is a native Kubernetes client for iPhone and iPad, with a full Mac app on the same core. It shows which pods each NetworkPolicy selects, answers "can A reach B?" step by step, and flags workloads a policy has cut off from DNS.

Get KubeGlance
#networkpolicy #dns #coredns #networking #troubleshooting

The Kubernetes dashboard that fits in your pocket

KubeGlance is a native Kubernetes client for iPhone and iPad — the real dashboard, not a companion — with a full Mac app on the same core. Pods, workloads, logs and events, straight from your kubeconfig.

Download KubeGlance