Record IAM errors as events

[pingles]

Although the Kiam server will currently record when its unable to retrieve credentials for a role (perhaps because the role doesn't exist, or isn't trusted by the node trying to AssumeRole) it means accessing Kiam logs.

It would be lovely to be able to see these errors reported as events for the Pod so that end-users can more easily track down issues.

[pingles]

I'm going to write my initial thoughts in case anyone wandering past reads and can help nudge my thinking along.

At the moment most interesting operations on behalf of Pods (determining the role, requesting credentials) are performed through exchanging role names, pod IPs etc. rather than a pod reference.

It feels like there are a couple of ways to change Kiam to support something like the event recording:

1. Agent reports errors when it is unable to respond successfully in one of it's metadata api impersonating handlers. This would probably need the Server changing its operations to include a ReportEvent operation, passing an IP to identify the pod.

2. Change the Server interface so that its chunkier and mirrors the metadata operations (e.g. GetRole, GetCredentials). The Agent HTTP interface would reduce to just calling that method and using the returned error to determine what happened. The Server could then store a reference to the Pod for other operations, recording events etc. I feel like this would simplify things down quite a lot, and could probably be added alongside the existing (although makes the existing methods somewhat deprecated/redundant).

In short- 1) maintains the current chatty form of rpc - GetRole, GetCredentials, IsAllowed etc., 2) reduces that to an operation per handler. 1 involves fewest changes to what's there, 2 is more drastic.

Having written it out I feel like 2 is probably the right thing to do.

[emmanuel]

This article from the Kubernetes blog may be useful. It contains some good explanation and code samples about using events to communicate from cluster components (e.g., kiam) to pods:
https://kubernetes.io/blog/2018/01/reporting-errors-using-kubernetes-events/

[pingles]

This is still really important to support- we occasionally have teams that get confused why their applications don't work and its because there are typos in role names etc.

I think this change to the server would help tidy a load of stuff up to. It'll mean adding a new set of operations to the server to eventually deprecate the current.

[pingles]

With #109 this gets closer for most operations. @Sambooo be good to go through what you and @DewaldV started to update so all stuff is tracked with the event recorder and get this closed!

[pingles]

The servers will now also need additional RBAC policy that lets them modify events. We should make sure this is documented in the #117 notes as well as in the current deployment manifests.

/cc @uswitch/cloud

[pingles]

@Sambooo could you add the notes/config for the updated RBAC permissions needed when you do your PR for the rest of this work please?