Add/remove certificates using API

[strarsis]

From what I understand, new certificates can be only set up using the fabio properties file.
When adding/removing apps or updating the PEM files, I would find it very helpful when fabio supports management of certificates using its API or even consul tags.

[magiconair]

Managing the certificates via the config file is a stop gap until the entire configuration moves to the backend to have a full cluster configuration. Also, hashicorp has Vault which is built for managing secrets. Both things are on my list of things to work on.

[strarsis]

If I want to use fabio in production now, how can I prevent issues when adding/removing certificates for existing apps?
Spawning up a second fabio container and shutting down the old one (blue/green deployment)?
Is there a config reload command for fabio?

[magiconair]

Interesting use case. Didn't take that into account. Yes, with the current setup you'd just update the configuration and restart which would interrupt existing connections. How often do the certificates change in your setup?

[magiconair]

No, there is no 'reload' command since by design this shouldn't be necessary... We'll find a solution.

[strarsis]

Thank you for your response.

For example: Let's Encrypt (LE) requires certificate renewal in intervals (90 days).
Also apps may be just added, new domains assigned to them,
in any of these cases there would be a small but existing downtime/disruption.

The vault PKI Secret Backend could be used to retrieve the PKI files:
https://www.vaultproject.io/docs/secrets/pki/index.html

For fabio authenticating itself against vault, using an Auth Backend,
the most straightforward for containers/apps would be App ID:
https://www.vaultproject.io/docs/auth/app-id.html

The main concern would be how to offer the configuration
for all these different Auth Backends in the fabio configuration.

As fabio is already opinionated insofar that it uses Consul and its protocol for service discovery,
adding vault as definite backend for obtaining the certificate keys/data would of course force the fabio operator to implement and maintain a vault server, too.

As kind of compromise, one could offer a pure HTTP API, similar to the existing one for adding routes to fabio ("overriding"), to allow assigning certificates to routes on-the-fly.
Then a kind of Vault-HTTP-adapter could be plugged in when vault should be used.

[magiconair]

Fabio will support different backends soon. The one for Google Compute Platform is in the works so there is definitely a tradeoff in using either Vault or some other configuration mechanism. Storing the certificates in the backend is simple at least for consul.

How would you store the certificates in consul? As PEM files without password or as P12 files with password? If you want password protection how does fabio get the password?

[strarsis]

Thank you for your answer.

As vault already handles the authentication of apps against itself in order to retrieve secrets,
wouldn't it be already sufficient to store the actual PEM files without passphrase?

Instad of a cert passphrase, fabio itself would auth against vault with a Auth Backend,
then vault will or will not give out the requested PEM files according to the permissions set for fabio in vault.

[magiconair]

Yes, that is correct if I use Vault for storing secrets. If you don't want that additional dependency then you get that unlock problem.

[strarsis]

So in nginx configuration for example, there is often an extra *_key file provided
which contains the passphrase
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_password_file

For haproxy config, the passphrase is always stripped first, otherwise it will ask for it at each restart.
http://blog.armbruster-it.de/2010/03/remove-the-passphrase-from-a-pkcs12-certificate/
The "password server" mentioned in this thread comes close to describing vault: http://comments.gmane.org/gmane.comp.web.haproxy/14943

It is often recommended that the passphrase is stripped from the cert when it is used in an automated way (like restarting a web server that uses it).
Shouldn't rather vault PKI backend offer an option to automatically strip + offer the unencrypted PEM to the authenticated vault client?

So for the non-vault API, there could be an extra parameter to the passphrase file, if needed,
when vault is used, a cert without passphrase is expected to be given to fabio.

Concerning the Auth Backend for fabio against vault:
With an app authenticating against vault, a token is generated by vault
and given back that then can be used for retrieving secrets from vault.

One way could be offering some kind of pre-auth script
so the admin is free to any use authentication backend of vault
which has to return a token to be used by fabio, each time when a cert has to be loaded or updated.
That pre-auth script should be ran as same user / in some system/container as vault itself.

[magiconair]

Storing the passphrase next to the cert isn't any benefit at all. You might as well just use the unlocked files. So the question then becomes how secure is your backend registry or the servers you're deploying to. I'm leaning towards this:

• fetch PEM files from URL (registry or otherwise). Retrieval via HTTPS would be nice but that's not how consul is usually deployed. least secure option. For dev or in trusted networks.
• fetch PKCS#12 files from URL (registry or otherwise) and provide passphrase during fabio startup either via properties file or on the command line. All certs would then require the same passphrase. Since someone who has access to the machine could also dump the memory to get to the passphrase properties should be good enough.
• use Vault which most likely also requires a key for fabio to unlock the vault

[strarsis]

I am really excited by this feature, together with HTTP2 support by Go 1.6 seeming soon coming out,
I can finally use fabio as elastic HTTP2/1 reverse proxy solution.

There is a similar reverse HTTP proxy, https://github.com/EmileVauge/traefik,
which from what I can find out about in its documentation,
only offers assigning certificate files via its config file,
it doesn't seem to be planned to implement dynamic addition/update during runtime,
neither vault backend support - so fabio would really have its edge here.

[strarsis]

Further thoughts:
Certs are assigned to common names which in turn are used by routes (route-prefixes).
domain1.com -> cert1.pem.
domain2.com -> cert2.pem

route1: domain1.com/whatever -> uses cert for domain1.com
route2: domain1.com/abc -> uses cert for domain1.com
route3: domain2.com/123 -> uses cert for domain2.com

Setting a cert that for an existing common name would override it (=reload),
this is necessary for certificate renewal.

1. Fabio API: Set certificate for CN domain1.com
2. Consul informs fabio about new service with prefix which uses CN domain1.com
   The other way around would mean that fabio would first serve only HTTP,
   when the certificate becomes available would start serving HTTPS, too.

[strarsis]

OK, so Traefik reads the CN of each passed certificate file
to determine automatically to what domains (and their routes) it applies:
https://github.com/emilevauge/traefik/blob/4e9ff45747f9172809065fb63c6d0baca782dad1/integration/fixtures/https/https_sni.toml

Something similar with HTTP API would be great for fabio.

[magiconair]

ok, so I can add multiple certificates to a single listener which are either statically or dynamically configured. This isn't really a code issue but a config issue. I need to think about this a bit more how I express this in the configuration. Once I've figured that out the implementation should be straightforward.