Metrics Apiservice "connect: no route to host"

[fefer]

I´m trying to deploy metrics to kubernetes and something really strange is happening, I have one worker and one master. I have the following pods list:

NAMESPACE     NAME                                              READY   STATUS    RESTARTS   AGE     IP               NODE                      NOMINATED NODE   READINESS GATES
default       php-apache-774ff9d754-d7vp9                       1/1     Running   0          2m43s   192.168.77.172   master-node               <none>           <none>
kube-system   calico-kube-controllers-6b9d4c8765-x7pql          1/1     Running   2          4h11m   192.168.77.130   master-node               <none>           <none>
kube-system   calico-node-d4rnh                                 0/1     Running   1          4h11m   10.221.194.166   master-node               <none>           <none>
kube-system   calico-node-hwkmd                                 0/1     Running   1          4h11m   10.221.195.58    free5gc-virtual-machine   <none>           <none>
kube-system   coredns-6955765f44-kf4dr                          1/1     Running   1          4h20m   192.168.178.65   free5gc-virtual-machine   <none>           <none>
kube-system   coredns-6955765f44-s58rf                          1/1     Running   1          4h20m   192.168.178.66   free5gc-virtual-machine   <none>           <none>
kube-system   etcd-free5gc-virtual-machine                      1/1     Running   1          4h21m   10.221.195.58    free5gc-virtual-machine   <none>           <none>
kube-system   kube-apiserver-free5gc-virtual-machine            1/1     Running   1          4h21m   10.221.195.58    free5gc-virtual-machine   <none>           <none>
kube-system   kube-controller-manager-free5gc-virtual-machine   1/1     Running   1          4h21m   10.221.195.58    free5gc-virtual-machine   <none>           <none>
kube-system   kube-proxy-brvdg                                  1/1     Running   1          4h19m   10.221.194.166   master-node               <none>           <none>
kube-system   kube-proxy-lfzjw                                  1/1     Running   1          4h20m   10.221.195.58    free5gc-virtual-machine   <none>           <none>
kube-system   kube-scheduler-free5gc-virtual-machine            1/1     Running   1          4h21m   10.221.195.58    free5gc-virtual-machine   <none>           <none>
kube-system   metrics-server-86c6d8b9bf-p2hh8                   1/1     Running   0          2m43s   192.168.77.171   master-node               <none>           <none>

When I try to get the metrics I see the following:

NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/50%   1         10        1          3m58s
free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$
free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$
free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$
free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$ kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$
free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$
free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$
free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$ kubectl top pods --all-namespaces
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

Lastly, I see the log (v=6) the output of metrics-server:

free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$ kubectl logs metrics-server-86c6d8b9bf-p2hh8  -n kube-system
I0206 18:16:18.657605       1 serving.go:273] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0206 18:16:19.367356       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 7 milliseconds
I0206 18:16:19.370573       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 1 milliseconds
I0206 18:16:19.373245       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 1 milliseconds
I0206 18:16:19.375024       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 1 milliseconds
[restful] 2020/02/06 18:16:19 log.go:33: [restful/swagger] listing is available at https://:4443/swaggerapi
[restful] 2020/02/06 18:16:19 log.go:33: [restful/swagger] https://:4443/swaggerui/ is mapped to folder /swagger-ui/
I0206 18:16:19.421207       1 healthz.go:83] Installing healthz checkers:"ping", "poststarthook/generic-apiserver-start-informers", "healthz"
I0206 18:16:19.421641       1 serve.go:96] Serving securely on [::]:4443
I0206 18:16:19.421873       1 reflector.go:202] Starting reflector *v1.Pod (0s) from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I0206 18:16:19.421891       1 reflector.go:240] Listing and watching *v1.Pod from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I0206 18:16:19.421914       1 reflector.go:202] Starting reflector *v1.Node (0s) from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I0206 18:16:19.421929       1 reflector.go:240] Listing and watching *v1.Node from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I0206 18:16:19.423052       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/nodes?limit=500&resourceVersion=0 200 OK in 1 milliseconds
I0206 18:16:19.424261       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0 200 OK in 2 milliseconds
I0206 18:16:19.425586       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/nodes?resourceVersion=38924&timeoutSeconds=481&watch=true 200 OK in 0 milliseconds
I0206 18:16:19.433545       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/pods?resourceVersion=39246&timeoutSeconds=582&watch=true 200 OK in 0 milliseconds
I0206 18:16:49.388514       1 manager.go:99] Beginning cycle, collecting metrics...
I0206 18:16:49.388598       1 manager.go:95] Scraping metrics from 2 sources
I0206 18:16:49.395742       1 manager.go:120] Querying source: kubelet_summary:free5gc-virtual-machine
I0206 18:16:49.400574       1 manager.go:120] Querying source: kubelet_summary:master-node
I0206 18:16:49.413751       1 round_trippers.go:405] GET https://10.221.194.166:10250/stats/summary/ 200 OK in 13 milliseconds
I0206 18:16:49.414317       1 round_trippers.go:405] GET https://10.221.195.58:10250/stats/summary/ 200 OK in 18 milliseconds
I0206 18:16:49.417044       1 manager.go:150] ScrapeMetrics: time: 28.428677ms, nodes: 2, pods: 13
I0206 18:16:49.417062       1 manager.go:115] ...Storing metrics...
I0206 18:16:49.417083       1 manager.go:126] ...Cycle complete
free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$ kubectl logs metrics-server-86c6d8b9bf-p2hh8  -n kube-system
I0206 18:16:18.657605       1 serving.go:273] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0206 18:16:19.367356       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 7 milliseconds
I0206 18:16:19.370573       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 1 milliseconds
I0206 18:16:19.373245       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 1 milliseconds
I0206 18:16:19.375024       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 1 milliseconds
[restful] 2020/02/06 18:16:19 log.go:33: [restful/swagger] listing is available at https://:4443/swaggerapi
[restful] 2020/02/06 18:16:19 log.go:33: [restful/swagger] https://:4443/swaggerui/ is mapped to folder /swagger-ui/
I0206 18:16:19.421207       1 healthz.go:83] Installing healthz checkers:"ping", "poststarthook/generic-apiserver-start-informers", "healthz"
I0206 18:16:19.421641       1 serve.go:96] Serving securely on [::]:4443
I0206 18:16:19.421873       1 reflector.go:202] Starting reflector *v1.Pod (0s) from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I0206 18:16:19.421891       1 reflector.go:240] Listing and watching *v1.Pod from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I0206 18:16:19.421914       1 reflector.go:202] Starting reflector *v1.Node (0s) from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I0206 18:16:19.421929       1 reflector.go:240] Listing and watching *v1.Node from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I0206 18:16:19.423052       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/nodes?limit=500&resourceVersion=0 200 OK in 1 milliseconds
I0206 18:16:19.424261       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0 200 OK in 2 milliseconds
I0206 18:16:19.425586       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/nodes?resourceVersion=38924&timeoutSeconds=481&watch=true 200 OK in 0 milliseconds
I0206 18:16:19.433545       1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/pods?resourceVersion=39246&timeoutSeconds=582&watch=true 200 OK in 0 milliseconds
I0206 18:16:49.388514       1 manager.go:99] Beginning cycle, collecting metrics...
I0206 18:16:49.388598       1 manager.go:95] Scraping metrics from 2 sources
I0206 18:16:49.395742       1 manager.go:120] Querying source: kubelet_summary:free5gc-virtual-machine
I0206 18:16:49.400574       1 manager.go:120] Querying source: kubelet_summary:master-node
I0206 18:16:49.413751       1 round_trippers.go:405] GET https://10.221.194.166:10250/stats/summary/ 200 OK in 13 milliseconds
I0206 18:16:49.414317       1 round_trippers.go:405] GET https://10.221.195.58:10250/stats/summary/ 200 OK in 18 milliseconds
I0206 18:16:49.417044       1 manager.go:150] ScrapeMetrics: time: 28.428677ms, nodes: 2, pods: 13
I0206 18:16:49.417062       1 manager.go:115] ...Storing metrics...
I0206 18:16:49.417083       1 manager.go:126] ...Cycle complete

Using the log output with v=10 I can see even the details of health of each pod, but nothing while running the kubectl get hpa or kubectl top nodes. Can someone give me a hint? Furthermore, my metrics manifest is:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
        args:
          - /metrics-server
          - --metric-resolution=30s
          - --requestheader-allowed-names=aggregator
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-insecure-tls
          - --v=6
          - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
            #- --kubelet-preferred-address-types=InternalIP
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        imagePullPolicy: Always
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      nodeSelector:
        beta.kubernetes.io/os: linux
        kubernetes.io/arch: "amd64"

And I have this too:

free5gc@free5gc-virtual-machine:~/Desktop/metrics-server/deploy$ kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  creationTimestamp: "2020-02-06T18:57:28Z"
  name: v1beta1.metrics.k8s.io
  resourceVersion: "45583"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: ca439221-b987-4c13-b0e0-8d2bb237e612
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2020-02-06T18:57:28Z"
    message: 'failing or missing response from https://10.110.144.114:443/apis/metrics.k8s.io/v1beta1:
      Get https://10.110.144.114:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.110.144.114:443:
      connect: no route to host'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

[serathius]

Get https://10.110.144.114:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.110.144.114:443 connect: no route to host
Points to your apiserver is having problems with reaching your metrics server pod. This is related not to metrics-server itself, but how your network is setup.

Usually it's caused by master node not running in container virtual network or some problems with virtual network itself.

I would suggest debugging why connection is failing. You can try curling metrics server address from your master node. https://10.110.144.114:443/apis/metrics.k8s.io/v1beta1

[fefer]

Get https://10.110.144.114:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.110.144.114:443 connect: no route to host
Points to your apiserver is having problems with reaching your metrics server pod. This is related not to metrics-server itself, but how your network is setup.

Usually it's caused by master node not running in container virtual network or some problems with virtual network itself.

I would suggest debugging why connection is failing. You can try curling metrics server address from your master node. https://10.110.144.114:443/apis/metrics.k8s.io/v1beta1

curling the address shows just:
curl: (7) Failed to connect to 10.110.144.114 port 443: No route to host

I cannot see any pod or something else related to this address. Do you know where this IP address is configured?

[serathius]

It should be either IP of your metrics-server or IP of Service

[fefer]

your

Thank you for your reply, it´s kind of strange, none of them match it. The service:

free5gc@free5gc-virtual-machine:~$ kubectl describe services
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP:                10.96.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         10.221.195.58:6443
Session Affinity:  None
Events:            <none>

The metrics-server:

free5gc@free5gc-virtual-machine:~$ kubectl get pod metrics-server-84d6975bf-r8jd4 -n kube-system -o wide
NAME                             READY   STATUS    RESTARTS   AGE    IP               NODE          NOMINATED NODE   READINESS GATES
metrics-server-84d6975bf-r8jd4   1/1     Running   0          116m   192.168.77.141   master-node   <none>           <none>

[serathius]

I would recommend to read about Api Server aggregation and what Network requirements it has. I don't think I can help more without deep understanding of your cluster and network setup.

[serathius]

Closing per Kubernetes issue triage policy

GitHub is not the right place for support requests.
If you're looking for help, check Stack Overflow and the troubleshooting guide.
You can also post your question on the Kubernetes Slack or the Discuss Kubernetes forum.
If the matter is security related, please disclose it privately via https://kubernetes.io/security/.

[MattFame]

hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet

the above properties in metrics-server deployment helped me to solve the issue together with --enable-aggregator-routing=true in kube-api-server and --kubelet-insecure-tls in metrics-server deployment.

[amitvashisttech]

Recently deployed Kubernetes on Azure & getting no route host error in kube-Api Server while connecting to metric server:

Cloud Platftom : Azure
Linux OS : Ubuntu 20.04 LTS
Kubernetes : v1.26.3
Network : Default Proxy
Metric Server : v0.6.3

Errors:

0329 22:27:12.419102       1 available_controller.go:527] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.110.183.33:443/apis/metrics.k8s.io/v1beta1: Get "https://10.110.183.33:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.110.183.33:443: connect: no route to host
E0329 22:27:24.191167       1 available_controller.go:527] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.110.183.33:443/apis/metrics.k8s.io/v1beta1: Get "https://10.110.183.33:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.110.183.33:443: connect: no route to host

Before:

root@myvm-0:~/# kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

Workaround:

    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
        - --kubelet-use-node-status-port
        - --metric-resolution=15s

      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true

After:

# kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
myvm-0   72m          3%     1317Mi          16%
myvm-1   21m          1%     837Mi           10%
myvm-2   34m          1%     887Mi           11%
#