Report Generation Fails with "lookup prometheus-server-exp: no such host"
Challenge
After enabling Veeam Kasten for Kubernetes reports, reports are not generated.
When running an on-demand report policy, the following error message is observed within the executor logs:
"message":"Post "http://prometheus-server-exp:80/k10/prometheus/api/v1/query\": dial tcp: lookup prometheus-server-exp on 192.0.0.10:53: no such host"
cause":{"message":"Failure in subordinate phase","function":"kasten.io/k10/kio/exec/phases/phase.(*queueAndWaitChildrenPhase).processGroup","linenumber":196,"file":"kasten.io/k10/kio/exec/phases/phase/queue_and_wait_children.go:196","fields":[{"name":"FailedSubPhases","value":[{"Phase":"","Err":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"Post "http://prometheus-server-exp:80/k10/prometheus/api/v1/query\": dial tcp: lookup prometheus-server-exp on 192.0.0.10:53: no such host"},"fields":[{"name":"query","value":"sum(round(increase(action_backup_skipped_overall{cluster=""}[24h])))"}]
Cause
This issue occurs when the Prometheus server hostname cannot be resolved via DNS, which prevents Veeam Kasten for Kubernetes pods (i.e., Executor pods) from capturing the Prometheus metrics. Since the Prometheus instance metrics cannot be collected, the Veeam Kasten for Kubernetes reports, which depend on those metrics, cannot be generated.
This may also occur when the DNS server is not running.
lookup prometheus-server-exp on 192.0.0.10:53: no such host
Solution
Cluster DNS Validation
Validate the cluster DNS by following this DNS troubleshooting guide.
The following commands can be used to help to check DNS functionality within the cluster:
- DNS debugging example pod YAML from kubernetes documentation:
- exec into the pod to run the nslookup command with the name <svcName.NamespaceName>:
Example of an issue with the cluster DNS:
> kubectl -n default exec -i -t dnsutils -- nslookup prometheus-server-exp.kasten-io Server: 10.245.0.10 Address: 10.245.0.10#53 ** server can't find prometheus-server-exp.kasten-io: NXDOMAIN command terminated with exit code 1
Troubleshoot DNS Service Functionality
The next steps are to check if the DNS service is running, whether endpoints are exposed, and restart the DNS deployment.
The commands below demonstrate troubleshooting with coredns as that is the built-in DNS service that comes with a Kubernetes cluster. If you are using a different DNS service, the troubleshooting procedure should be similar, check documentation to be sure.
- Check if DNS pods are running filtering by label kube-dns:
- Check if DNS endpoints are discovered:
- Try restarting DNS services (rollout):
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.
Spelling error in text
Thank you!
Your feedback has been received and will be reviewed.
KB Feedback/Suggestion
This form is only for KB Feedback/Suggestions, if you need help with the software open a support case
Thank you!
Your feedback has been received and will be reviewed.
Oops! Something went wrong.
Please, try again later.