Tiller pods can't connect to k8s apiserver

[willise]

logs:

(kube-master) tiller # kcm logs tiller-deploy-3354596499-4q16f
Cannot initialize Kubernetes connection: Get http://localhost:8080/api: dial tcp [::1]:8080: getsockopt: connection refused

This issue has been raised in #1591 #1791.

In #1791, there gets a resolution(thank @iamzhout) of adding KUBERNETES_MASTER in pods env but nobody explains the root cause of the issue.

I installed the k8s cluster manually and it seems that tiller pod can't get KUBERNETES_MASTER from env as this item doesn't exist in the env at all.

No docs are found about the relation between k8s cluster and the configuration. I don't think it is a good idea to add the env item manually(and how to helm init with the configuration???)

Is there anybody who can offer the help?

Thanks anyway (^.^).

[technosophos]

@willise You might want to ask about this in the Slack channel if possible. It is very likely a configuration issue with Kubernetes, and people there may be able to help quickly.

[bacongobbler]

Agreed. This looks to be more of a misconfigured kubernetes cluster causing the KUBERNETES_SERVICE_HOST environment variable to either be pointing in the wrong location or it doesn't exist at all. Tiller needs that envvar in the pod to connect to the kubernetes apiserver.

I don't think there's anything actionable here in helm itself, save perhaps some docs clarification on how tiller communicates with the apiserver.

[willise]

@technosophos Ooops... Kind of your reminding and I'll note that.

[willise]

@bacongobbler KUBERNETES_SERVICE_HOST exists in env indeed but it's value is a cluster IP like 10.254.0.1. If I don't set KUBERNETES_MASTER, tiller is unable to connect apiserver port 8080. Only if it is set to master physical address like 192.168.56.101, it's ok.

So the problem may be tiller can't get the correct apiserver address from k8s when tiller initializes the k8s apiserver connection.

[allamand]

I've got a similar error where tiller retrieve the correct ip and port but can't connect due to tls verification (i use skip-tls-verify on my kubernetes.

2017-05-30T10:52:56.35985785Z Cannot initialize Kubernetes connection: Get https://10.100.200.1:443/api: x509: cannot validate certificate for 10.100.200.1 because it doesn't contain any IP SANs

is there a way to ask tiller to skip this verification ?

[penfree]

I meet the same problem, it was due to KUBECONFIG is not default. so you shoud specify KUBECONFIG env to the right position.

[mvernimmen]

@bacongobbler I'm having the same issue here, while KUBERNETES_SERVICE_HOST is set correctly:

$ kubectl exec -it tiller-deploy-1490305151-6jq17 -n integrations-test -- /bin/sh
/ # env | grep -i kube
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_SERVICE_HOST=10.96.0.1

To fix the problem I set the KUBERNETES_MASTER in the running tiller pod and tried again, but when doing an install, the problem remains:

$ kubectl logs tiller-deploy-1490305151-6jq17 -n integrations-test
...
[tiller] 2017/12/28 13:34:09 preparing install for rmlater
[storage] 2017/12/28 13:34:09 getting release history for "rmlater"
[storage/driver] 2017/12/28 13:34:09 query: failed to query with labels: Get http://localhost:8080/api/v1/namespaces/integrations-test/configmaps?labelSelector=NAME%3Drmlater%2COWNER%3DTILLER: dial tcp 127.0.0.1:8080: getsockopt: connection refused
[tiller] 2017/12/28 13:34:09 failed install prepare step: Get http://localhost:8080/version: dial tcp 127.0.0.1:8080: getsockopt: connection refused

And trying the install with --kube-context integrations-test or with KUBECONFIG set to the path of my .kube/config file did not resolve the problem either.

According to the kubernetes manual, the best practice is to resolve the kubernetes (fqdn) hostname (https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod), is that not what is being done here?
Why is the KUBERNETES_SERVICE_HOST not being used to access the API?
This is all with version 2.7.2

I'm probably missing something obvious, as are some of the other people having this problem. Could you, or someone else please have a look into this?

[mvernimmen]

I have found 1 possible cause for this problem.
On a cluster set up with kubespray, running version 1.8.3, helm init and helm install work when executed by a user when deploying to a specific namespace. But only when the serviceaccount name is 'tiller'. When using serviceaccount name 'tiller-integrations-acc', the 127.0.0.1 error appears.
I've pasted my steps here: https://pastebin.com/TwCypx4Q
Did I do it wrong or did I run into a bug?

However, this did not solve the problem on a cluster set up using kubeadm and running k8s version 1.7.8, there the 127.0.0.1 API problem remains despite using the serviceaccount name 'tiller'. So perhaps there are multiple possible causes for the error message.

[mattus]

Had something very similar to this, our eventual fix was to ensure automountServiceAccountToken on the ServiceAccount running tiller was set to true. We were creating the service account with terraform which was defaulting to false for reasons discussed in hashicorp/terraform-provider-kubernetes#38.

Manually creating the service account with kubectl does not set this value (it defaults to true when unset).

[mvernimmen]

Hi Mattus, thank you for suggesting that. I tried it, but unfortunately for me it doesn't fix the problem. installing charts when using a service account name other than 'tiller' results in helm not being able to find 'tiller'.

[bacongobbler]

According to #3816 and a comment in #1791, it appears that the missing environment variable that is injected from cloud providers and assumed to be present as per client-go is KUBERNETES_MASTER, not KUBERNETES_SERVICE_HOST. I'm not sure what the resolution is, but this assumption is embedded in client-go and is consistent across all the major cloud providers, so it's probably something missing in the bare metal cluster setup guide which breaks this assumption. :/

tl;dr this is a bare metal cluster doc issue, not necessarily something wrong with helm's assumptions about the running environment.

[fossxplorer]

I'm posting the comment here as well from issue #3870:
It seems to be an issue with the running Tiller pod:
[root]# docker logs --tail 1 k8s_tiller_tiller-deploy-686c785c58-cl22s_kube-system_34734fe7-3e4c-11e8-a4dd-fa163e87b224_0
[storage/driver] 2018/04/12 13:02:59 list: failed to list: Get http://localhost:8080/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: dial tcp 127.0.0.1:8080: connect: connection refused

The above was triggered by helm on my workstation so the command is run remotely against the cluster:
$ helm list
Error: Get http://localhost:8080/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: dial tcp 127.0.0.1:8080: connect: connection refused.

Is Tiller supposed to be running something (local proxy of some kind that uses KUBE* env variables to forward traffic?) locally on port 8080?
As far as i can see, there is nothing listening on port 8080 within the Tiller pod (which makes sense with the error message from Helm):
docker exec -i k8s_tiller_tiller-deploy-686c785c58-cl22s_kube-system_34734fe7-3e4c-11e8-a4dd-fa163e87b224_0 netstat -tlnup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::44134 :::* LISTEN 1/tiller
tcp 0 0 :::44135 :::* LISTEN 1/tiller

Is there anything else i might have missed and should check?