Report #146
[bug\_fix] Kubernetes service DNS resolution failure
Start a debug pod in the same namespace and run 'nslookup kubernetes.default' and 'nslookup ..svc.cluster.local'. If internal names fail, check that CoreDNS pods in kube-system are Running and Ready and that no NetworkPolicy blocks egress UDP/TCP port 53 to the kube-dns service. If external names fail, inspect the CoreDNS Corefile forward directive and node-level upstream DNS. If resolution is slow, address ndots search-domain amplification or the conntrack UDP race with single-request-reopen.
Journey Context:
An application log shows 'could not resolve host my-service' or connection timeouts to other services by name. From inside a debug pod, cat /etc/resolv.conf shows nameserver pointing at the kube-dns ClusterIP and search domains default.svc.cluster.local svc.cluster.local cluster.local. nslookup kubernetes.default fails, which means cluster DNS is broken rather than the target service being missing. The next checks are kubectl get pods -n kube-system -l k8s-app=kube-dns to confirm CoreDNS is running, kubectl logs on CoreDNS for errors, and kubectl get networkpolicy in the affected namespace because a default-deny policy often blocks UDP/TCP port 53 to kube-system. If CoreDNS is healthy and no policy blocks it, external failures point to the Corefile forward directive and the node's upstream resolver. When DNS works but is intermittently slow, the cause is often the glibc conntrack race for simultaneous A/AAAA queries or ndots:5 causing four lookups per external name; fixes include NodeLocal DNSCache, lowering ndots, or adding single-request-reopen.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-12T18:36:19.511171+00:00— report_created — created