Intermittent connectivity failures

[JethroMV]

Hi, we love kiam it's great!
We're deploying it on our AWS KOPS K8s cluster using the following settings:

agent:
  gatewayTimeoutCreation: 20s
  timeout: 20s
  log.level: debug
  host:
    interface: weave
    iptables: true
server:
  gatewayTimeoutCreation: 20s
  timeout: 20s
  log.level: debug
  assumeRoleArn: arn:aws:iam::123456789:role/my-kiam-server
  nodeSelector:
    kubernetes.io/role: master
  tolerations:
  - key: "node-role.kubernetes.io/master" 
    effect: NoSchedule 
  sslCertHostPath: /etc/ssl/certs

Everything was working well for several weeks but we've started to notice that individual nodes will occasionally fail to connect. Restarting the kiam-agent pod has never gotten it unstuck. Neither has restarting the kiam-server.
Once a node has turned bad we've had to drain/delete the node, and Kops gives us a new one which works most of the time. But sometimes, like today, all new nodes fail too.

Any help debugging this would be amazing!

We are using the latest version from helm. Ie we install using helm install kiam uswitch/kiam --namespace kiam --values kiam-values.yaml.
Kops Version: 1.14.0.
kubernetes Version: 1.16
Container Runtime Version: docker://18.6.3
Kubelet Version: v1.12.10

Example server logs. Sometimes these look healthier.
kiam-server.log
It will often get to a (I assume healthy) state of repeating: kiam-server-p2wmk kiam-server {"level":"info","msg":"found role","pod.iam.role":"arn:aws:iam::630521468456:role/jura-stereotypes-read","pod.ip":"100.121.0.4","time":"2020-01-09T13:54:54Z"}

Example agent logs case 1: the most common case is an error on starting up
kiam-agent1.log

Example agent logs case 2: in the second most common case the logs get further
kiam-agent2.log

The main error seems to be:
{"level":"fatal","msg":"error creating server gateway: error dialing grpc server: context deadline exceeded","time":"2020-01-09T13:28:29Z"}

Sometimes it's been working, but it seems to be getting worse. We haven't changed our pod that is annotated to use a role.
Thanks for any help debugging this!

[caiohasouza]

+1

[johnmccabe]

Have you tried using the latest 3.5 release, it includes a gRPC keepalive change #337 which will likely help you.

[JethroMV]

Have you tried using the latest 3.5 release, it includes a gRPC keepalive change #337 which will likely help you.

Thanks for the suggestion, do you have instructions on how to do that? Should I be building from source?
I've been using helm, which only has up to version 3.4.

[johnmccabe]

You could set the agent.image.tag and server.image.tag at install time (see the Helm README.md) to use the 3.5 tagged image.

I raised a PR to update the image in Helm a short while ago.

[JethroMV]

Thanks, but unfortunately only 2/6 agents with kiam:v3.5 have connected successfully.
kiam-logs.txt
I will keep having a play with the new version to see if I can get anything working.

[Joseph-Irving]

Have you tried testing if those nodes can connect to the Kiam server? It sounds like you're having some kind of network connectivity problems if some nodes can just never talk to it, is it possible you have some conflicting IP-tables rules or network policy which is sometimes blocking traffic?

If it were a grpc keep-alive issue restarting the kiam agent would've solved the problem. The fact that the only way to fix it is to spin up a new node, implies that there is something odd about that node itself.

[JethroMV]

I can ping from each kiam-agent Pod to the kiam-server OK. And from kiam-server out to the public internet. I can reliably download our AWS resources from the master node, can't spot any problems although maybe aws s3 cp is smoothing problems over.
Will dig into our network policies, it was crated with Kops and we didn't intend to change anything from the default, but maybe we did or the defaults aren't sensible for this.
If that fails, I'll build a new cluster and try there.
Thanks for the help

[JethroMV]

A new cluster works, same versions of everything (I used kiam 3.5 again).
Digging into network policies I didn't find anything, but further digging showed the old cluster seemed to timeout about 10% of DNS requests. The new cluster doesn't. Perhaps that was enough to cause the problems, not sure.
Thanks for the help all, it's much appreciated!

[caiohasouza]

@JethroMV how you identify the DNS network requests problem ?

[JethroMV]

@JethroMV how you identify the DNS network requests problem ?

I ssh'd onto each Node of the cluster kubectl get nodes ssh -i my.pem admin@ip and ranwhile true; do nslookup s3.amazonaws.com; done which showed that it would occasionally hang, and wget http://s3.amazonaws.com would occasionally fail with wget: bad address 's3.amazonaws.com'.
Only about 1 in 10, and it seems to come and go at that... it seems newer nodes succeed more often.
ping 8.8.8.8 works reliably, which suggests to me the link is good but DNS is bad.
I feel like it was worse running those test commands from within a Pod on a node eg kuebctl exec -it mypod-123 -- /bin/sh -c "nslookup s3.amazonaws.com" compared to directly on the node but wasn't sure.

I also found this useful: https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/. I didn't find the root problem but have been able to work around it using the new cluster in my case. Worth a read for sure. I'm worried it might come back with time, but will see. It helped to narrow the problem by checking the nslookup tests from a Pod, and then from the Node itself, to rule out if it was a Node>Internet problem or a internal-K8s problem.
Disclaimer: I'm not a networking guy so this is all best guesses.

[caiohasouza]

@JethroMV right, in you new cluster, when you start a new node, the kiam-agent fail to start few times? In my case the main problem is that the kiam-agent on new nodes always fail few times on start (3 or 4 times), after this they start and works.

[JethroMV]

@JethroMV right, in you new cluster, when you start a new node, the kiam-agent fail to start few times? In my case the main problem is that the kiam-agent on new nodes always fail few times on start (3 or 4 times), after this they start and works.

We also always got a couple of restarts for the kiam-agents, I assume that's because the kiam-server is still starting up.
But the problem I saw described in this issue was continuous restarts, I think I reached 800.
Unfortunately, the new cluster has started to display the same problem. I'll try to get a reproducible test case if I can.

[caiohasouza]

@JethroMV i understand, Ok, thank you!