Unable to access new AWS metadata api

[snstanton]

Amazon recently updated their metadata api. When I attempt to follow the basic instructions to retrieve metadata as described here, kiam rejects the request with:
request blocked by whitelist-route-regexp "^$": /latest/api/token

[tbp105]

For reference, this is the IMDSv2 version of the metadata api that needs support.

[nicolasbelanger]

Any idea on an ETA for this? We have to hold back on aws-sdk-js updates until then...

FYI, 2.575.0 (19 Nov 2019) is the version where the new flow was introduced. aws/aws-sdk-js@4ed5568#diff-2e7c25b2cedb392fadaae6bc592929d6R106-R126

We manage to keep errors off by sticking to 2.573.0...

[rhysemmas]

We don't yet have an ETA for adding support for the new IMDSv2 token path into Kiam, but you should be able to allow the requests through in the meantime by adding the token path to the Kiam agent whitelist regex.

[rbvigilante]

you should be able to allow the requests through in the meantime by adding the token path to the Kiam agent whitelist regex.

It looks like IMDSv2 is set up in a way that prevents this working 😞. There's a lot of context in this kube2iam issue and this aws-sdk-ruby one, but the summary seems to be:

• IMDSv2 is meant to protect, among other things, against "open" reverse proxies, where a (say) squid proxy can be set up to provide external access to the metadata API. The way it does this is to blanket refuse any request that contains an X-Forwarded-For header, regardless of what's actually in the header (the announcement mentions this explicitly)
• The Golang httputil.ReverseProxy is set up in such a way as to always set the X-Forwarded-For header (because that's generally correct behaviour for a reverse proxy).
• So, any request made to IMDSv2 will be refused with a 403 when it's made through Kiam (or kube2iam, for that matter).

All official language SDKs have been told they must adopt IMDSv2; we're seeing issues with recent versions of the Ruby and Golang SDKs too.

This is causing several of the teams I work with a bit of pain, and I'd be super happy to attempt a PR to fix it (if you're open to that), but I can't see a nice way to go about it (i.e, one that doesn't involve implementing a custom reverse proxy).

[Joseph-Irving]

This does seem like a rather awkward change, we will definitely need a solution to this as I'm sure there will be reasons why people have to upgrade their SDK.
I haven't really studied the problem enough to have any concrete ideas how we could solve the problem in Kiam.
Very much open to any ideas @rbvigilante

[rbvigilante]

From what I can see, it appears that the only way to really get around this issue is to stop adding an X-Forwarded-For header. This feels a bit shifty (it's deliberately avoiding the standard pattern for proxies and is concealing the origin of requests), but I'm not sure there's another way around. I doubt Amazon will budge on this particular feature of the metadata service; people have already raised more or less these precise concerns with them and been largely ignored.

I've poked around the Internet looking for alternative proxies in Golang, but (I guess because the httputil one is both good and super easy to use), there don't appear to be any currently-supported options.

My team and I talked (not very seriously) about hooking up a second proxy around Kiam (to forward the request to the metadata service after removing the X-Forwarded-For header; I'm sure one of the off-the-shelf proxies can do that), but we all agreed that felt like an inefficient kludge.

I doubt we'll have much success petitioning the Golang maintainers to make the X-Forwarded-For header in the httputil proxy optional - the behaviour as it stands is exactly what a reverse proxy is meant to do.

I guess where I'm going with this is that I think the only real way to get around this is to basically reïmplement (or just, like, copy wholesale) the httputil proxy, but leave out the bit where the X-Forwarded-For header is introduced. I don't really like anything about this, but I'm really struggling to come up with a better option.

It looks like basically everything else about the API remains the same after that PUT request to /latest/api/token - there's just a header with a token included in every request. So long as that token is passed by the proxy, I don't think anything else would need to be done to support IMDSv2 (that said, I don't have a strong understanding of what Kiam does when it overrides the role a pod asks for).

[tbp105]

When kiam is acting as a client, it can just use the IMDSv2 protocol as intended, it has actual direct access to the real metadata server so it'll just work. (The kiam server does this)

When it is acting as the server (the kiam agent), it'll need to implement the /latest/api/token handler and check the value in the "x-aws-ec2-metadata-token" header for subsequent metadata requests. A JWT based token would be a fairly straightforward implementation since kiam already has a cert it could sign them with. Other than the expiry time, I don't think any data really needs to be in the jwt, it isn't intended to replace the role matching or namespace filtering.

The communication between the agent and server does not need to change.

[rbvigilante]

When it is acting as the server (the kiam agent), it'll need to implement the /latest/api/token handler and check the value in the "x-aws-ec2-metadata-token" header for subsequent metadata requests. A JWT based token would be a fairly straightforward implementation since kiam already has a cert it could sign them with. Other than the expiry time, I don't think any data really needs to be in the jwt, it isn't intended to replace the role matching or namespace filtering.

If Kiam produces its own tokens, what happens if a pod holding a token Kiam issued goes on to try to talk to a part of the metadata service that Kiam is proxying through to the IMDS?

[rbvigilante]

I would propose something like this:

• Stop adding X-Forwarded-For from all proxied requests
• PUT */api/token is always proxied to the metadata API transparently
• Kiam doesn't actually need to be given a token in order to hand out a role (for the reasons @tbp105 pointed out above; internally it can use the API as intended). In order to maintain the security benefits of IMDSv2, the agent (running on a worker node) could possibly make a request to /latest/iam/credentials with the token provided by the client; if that request is denied, it could refuse to provide the pod's role (use the IMDS as an authentication provider).
• Any requests which would otherwise be proxied to the IMDS are proxied though with their tokens unchanged.

[rbvigilante]

I was discussing this issue with some other people inside Redbubble today, and found out that it might be possible to remove the X-Forwarded-For header within the Go ReverseProxy by removing the RemoteAddr from the request in a custom Director. I'll see if I can set up a pull request to fix at least this portion of the problem tomorrow.