Vault token should not require 'root' or 'sudo' privileges

[far-blue]

I'm investigating the new integration options for fabio with cert stores, esp. vault (btw. great work, really looking forward to using it!) and I'm struggling with Vault Tokens.

I've created an orphan token with a limited policy but when I try to use it with Fabio I'm getting the following:

2016/07/26 12:35:57 [FATAL] vault: client: Error making API request.

URL: POST http://vault-server:8200/v1/auth/token/create
Code: 400. Errors:

• root or sudo privileges required to create orphan token

This would suggest Fabio needs a token with a root policy. I'm not sure I'm happy about having a token with root-level access floating about my config files and live environments.

Is there a reason fabio needs a root token? Am I doing something wrong?

[magiconair]

I'll have a look.

[magiconair]

I don't think this is a hard requirement. I've created the fabio token as an orphaned token which apparently requires these privileges. I'll check what the best approach is and make this configurable if necessary. If you have suggestions on how to set this up feel free to comment.

[far-blue]

Maybe it would help to understand why you are trying to create an orphan token?

My assumption was that I would be responsible for supplying a token and keeping it renewed and all Fabio would do was use the token it was given. If you prefer to generate short-lived tokens based on the supplied token on startup and then cycle them internally for your own reasons then I think that will still work.

Orphaning the token would mean it wasn't revoked when the parent token was revoked which is probably not the behaviour you want - and, of course, orphaning can only be done by root.

In fact, the token I'm supplying is itself orphaned as I didn't want it related to my login token but I think that's something for the Vault admin to deal with rather than Fabio :)

[magiconair]

The rationale is that you provide a short-lived (e.g. 30s) token to fabio during startup so that fabio could generate its own token which it continues to use and extend as long as it is running. Otherwise, anybody who can inspect the environment via the /proc filesystem or the command line arguments would be able to re-use the token.

However, it is perfectly possible that I misunderstood something and that this is achievable without creating an orphan token. I remember having a discussion on the Vault approach fabio takes with @jeffrai - the Vault maintainer. I'll dig into the docs again.

[far-blue]

To combat the problem of the token being visible you could make use of the new 'wrapping' capability in Vault. With this facility, the token supplied to Fabio would be a one-use token and fabio would query Vault using the unwrap endpoint to fetch the actual token to use. Other than this extra step the token will behave as normal.

The problem with this approach is that a new wrapped token would be needed every time Fabio is started meaning some form of automated script would prob. be needed to do it and that script would then need its own token in order to generate the wrapped token. I guess the advantage here is that this script would be short-lived and while you'd need to protect the token on disk you'd not see it in memory. After all, we are used to storing private cert keys on disk and if the token was stolen then it would only give access to the same private certs that could have been stolen from disk with the same level of access as the token - and the token can at least be revoked easily!

The other problem is one of renewing the token because this is another task prob. best automated and so, again, another script needing a token.

I'd be interested in what @jeffrai suggests :)

[magiconair]

I think I prefer the approach where fabio gets the ability to create and manage its own token since you then don't have to worry about renewals and visibility. In your approach you'd still need a short-lived token to bootstrap fabio only for fabio to re-use the same token and move token renewal outside of fabio.

One question is whether that requires the creation of an orphaned token. Another is what happens when the parent token expires? This should invalidate the child tokens as well. And I also need to check whether I deal with the expired token the right way, i.e. invalidate the cert store instead of treating it as "connection error" which means that the certs remain valid.

In our environment we don't restart fabio for days, weeks or even months. It just runs and we forget about it.

As far as I recall the conversation with @jeffrai during HashiConf EU he suggested the approach of the short-lived startup token. But at that time Vault 0.6.0 was just out and the guy was quite popular. Not sure if we had enough time to talk about it in-depth.

[magiconair]

Hmm, this won't work. If you pass a short-lived token to fabio and create a renewable child token with it then the child will expire when the parent expires unless I'm missing something.

I've created a script which simulates the behavior:

https://gist.github.com/magiconair/d0a5f8bf8605f1c21064132edd7344e1

How should this work?

[far-blue]

I don't think fabio needs to create child tokens and I don't think it should create orphan tokens. Fabio's token is basically a password or an oauth type key and so it would make sense for an external system to create a token with specific permissions (i.e. policy) and hand it to fabio to use.

Firstly, it makes sense for the token to expire if not used. But it looks like there is support for self-renewal of a token (assuming the token, when created, was set to allow renewal and the policy is configured to allow it as well). See the '/auth/token/renew-self' POST endpoint in the docs (link at the end of the comment). I'm not sure how you can find out the expiration of the token given to fabio but when you renew-self you get back info on the lease duration which you could log in consul so as to renew again in good time (and rinse/repeat).

I don't think it's a good idea to create orphan child tokens because whoever / whatever gave the token to fabio should retain the ability to revoke access. If orphan child tokens are created the revocation process is significantly harder or possibly even impossible.

To cover your concerns about the token being provided to fabio in a way that is sniffable, I suggest support for wrapped tokens. The token provided to fabio on startup would be a one-shot wrapped token which fabio would use to fetch the actual token. Pretty much anything in Vault starting v0.6 can be wrapped and this functionality makes use of the cubbyhole secret backend.

A final and somewhat different approach would be to skip the tokens entirely and use the AppID auth backend to authenticate. Here, you could accept the appId via an env var and calculate the userId through some machine-specific value (mac address is suggested but not sure that's sensible in a container). You can also lock the appId/userId combinations down to a certain CIDR. Authentication is linked to a policy and so you still end up with a token with ACLs which you can, I assume, renew as needed as long as fabio is running. Arguably, however, this authentication could be the job of a pre-launch helper script which does the login and fetches a wrapped token which is then supplied to fabio as in the previous paragraph.

https://www.vaultproject.io/docs/auth/token.html
https://www.vaultproject.io/docs/secrets/cubbyhole/index.html
https://www.vaultproject.io/docs/auth/app-id.html

[far-blue]

Thinking about it, maybe @jeffrai was referring to these 'wrapped' tokens when he talked about the short-lived startup tokens.

[magiconair]

Possibly. OK, then fabio would get a token, unwrap it and renew it on a regular basis. I think that makes sense.

[magiconair]

I think I have something working but this needs a bit more refactoring since the current code tries to unwrap the token twice which won't work.

[magiconair]

@far-blue I've pushed a patch that changes the way fabio integrates with vault. It will use the provided token instead of creating a new one and will transparently unwrap a wrapped token. fabio still tries to refresh the token on every refresh cycle but it will no longer fail if it can't. However, it logs a warning on every failure which has the potential of filling up the logs quickly if provided with a non-renewable token. I've also updated the integration test to use a policy to limit the required capabilities.

I think this is starting to look better than the original approach but I need to test the edge cases a bit more. It would be great if you could test this setup.

The required policies are (assuming that you configured your cert store with secret/fabio/cert):

  path "secret/fabio/cert" {
    capabilities = ["list"]
  }

  path "secret/fabio/cert/*" {
    capabilities = ["read"]
  }