Add parameter to vault token renewal

[deuch]

Fabio need to have a parameter in the fabio.properties to let the user set the renewal interval for the Vault Token. It's 1h actually and it's too long :-) We've issue because our tokens have a 20mn lease.

[magiconair]

Sure. Was expecting that this would happen at some time.

[magiconair]

@deuch I've pushed a change that should make the token renew interval configurable. You have to set the renewtoken option in the proxy.cs statement. Could you check whether that solves the problem for you?

[deuch]

Hello,

I've made the test and it doesn't seem to work, fabio doesn't want to start, telling me that the port is already used ... Stop fabio, kill every instance, check with netstat (root) and nothing is listening on that port.

Revert back to 1.4.3 and it works fine (with exact same properties file) ...

Please find the logs :

2017/04/27 09:31:25 [INFO] Runtime config
{
    "Proxy": {
        "Strategy": "rr",
        "Matcher": "prefix",
        "NoRouteStatus": 404,
        "MaxConn": 10000,
        "ShutdownWait": 0,
        "DialTimeout": 30000000000,
        "ResponseHeaderTimeout": 0,
        "KeepAliveTimeout": 0,
        "FlushInterval": 1000000000,
        "LocalIP": "192.163.167.143",
        "ClientIPHeader": "",
        "TLSHeader": "",
        "TLSHeaderValue": "",
        "GZIPContentTypes": null
    },
    "Registry": {
        "Backend": "consul",
        "Static": {
            "Routes": ""
        },
        "File": {
            "Path": ""
        },
        "Consul": {
            "Addr": "dckdevdck075.mydomain.com:8543",
            "Scheme": "https",
            "Token": "xxxxxxxxxxxxxxxxxxxxxxxxx",
            "KVPath": "/l7demo1/config",
            "TagPrefix": "urlprefix-",
            "Register": true,
            "ServiceAddr": ":9998",
            "ServiceName": "l7demo1",
            "ServiceTags": null,
            "ServiceStatus": [
                "passing"
            ],
            "CheckInterval": 1000000000,
            "CheckTimeout": 3000000000,
            "CheckScheme": "http",
            "CheckTLSSkipVerify": false
        },
        "Timeout": 10000000000,
        "Retry": 500000000
    },
    "Listen": [
        {
            "Addr": "dckdevdck075:11443",
            "Proto": "tcp+sni",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "RenewToken": 0,
                "Header": null
            },
            "StrictMatch": false
        },
        {
            "Addr": "dckdevdck075:12443",
            "Proto": "https",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "CertSource": {
                "Name": "ssl-vault",
                "Type": "vault",
                "CertPath": "secret/fabiodemo-001/certs",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 3000000000,
                "RenewToken": 900000000000,
                "Header": null
            },
            "StrictMatch": false
        }
    ],
    "Log": {
        "AccessFormat": "$remote_host - - [$time_common] \"$request\" $request_proto $response_status $response_body_size $upstream_addr $upstream_host $upstream_port $upstream_request_url $upstream_request_uri",
        "AccessTarget": "stdout",
        "RoutesFormat": "delta"
    },
    "Metrics": {
        "Target": "statsd",
        "Prefix": "{{clean .Hostname}}.{{clean .Exec}}",
        "Names": "{{clean .Service}}.{{clean .Host}}.{{clean .Path}}.{{clean .TargetURL.Host}}",
        "Interval": 30000000000,
        "GraphiteAddr": "",
        "StatsDAddr": "184.44.245.74:8125",
        "Circonus": {
            "APIKey": "",
            "APIApp": "fabio",
            "APIURL": "",
            "CheckID": "",
            "BrokerID": ""
        }
    },
    "UI": {
        "Listen": {
            "Addr": ":9998",
            "Proto": "http",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "RenewToken": 0,
                "Header": null
            },
            "StrictMatch": false
        },
        "Color": "light-green",
        "Title": ""
    },
    "Runtime": {
        "GOGC": 800,
        "GOMAXPROCS": 1
    }
}
2017/04/27 09:31:25 [INFO] Version  starting
2017/04/27 09:31:25 [INFO] Go runtime is go1.8.1
2017/04/27 09:31:25 [INFO] Sending metrics to StatsD on 184.44.245.74:8125 as "dckdevdck075.fabio"
2017/04/27 09:31:25 [INFO] Sending metrics to StatsD on 184.44.245.74:8125 as "dckdevdck075.fabio"
2017/04/27 09:31:25 [INFO] Setting GOGC=800
2017/04/27 09:31:25 [INFO] Setting GOMAXPROCS=1
2017/04/27 09:31:25 [INFO] consul: Connecting to "dckdevdck075.mydomain.com:8543" in datacenter "dcfabio"
2017/04/27 09:31:25 [INFO] Admin server listening on ":9998"
2017/04/27 09:31:25 [INFO] Waiting for first routing table
2017/04/27 09:31:25 [INFO] consul: Using dynamic routes
2017/04/27 09:31:25 [INFO] consul: Using tag prefix "urlprefix-"
2017/04/27 09:31:25 [INFO] consul: Watching KV path "/l7demo1/config"
2017/04/27 09:31:25 [INFO] consul: Manual config changed to #1
2017/04/27 09:31:25 [INFO] TCP+SNI proxy listening on dckdevdck075:11443
2017/04/27 09:31:25 [INFO] consul: Health changed to #21965
2017/04/27 09:31:25 [INFO] Config updates
+ route add l7demo1-tomcatfabiosni tomcat-sni.mydomain.com/ http://dckdevdck012:30005/
+ route add l7demo1-tomcatfabio tomcat-fabio.mydomain.com/ https://dckdevdck012:30030 opts "proto=https tlsskipverify=true"
2017/04/27 09:31:25 [INFO] consul: Registered fabio with id "l7demo1-dckdevdck075-9998"
2017/04/27 09:31:25 [INFO] consul: Registered fabio with address "192.163.167.143"
2017/04/27 09:31:25 [INFO] consul: Registered fabio with tags ""
2017/04/27 09:31:25 [INFO] consul: Registered fabio with health check to "http://[192.163.167.143]:9998/health"
2017/04/27 09:31:25 [INFO] HTTPS proxy listening on dckdevdck075:12443
2017/04/27 09:31:25 [INFO] Writing access log to stdout
2017/04/27 09:31:25 [INFO] Using routing strategy "rr"
2017/04/27 09:31:25 [INFO] Using route matching "prefix"
2017/04/27 09:31:25 [FATAL] listen: Fail to listen. listen tcp 192.163.167.143:12443: bind: address already in use
2017/04/27 09:31:25 [FATAL] accept tcp 192.163.167.143:12443: use of closed network connection
2017/04/27 09:31:25 [INFO] consul: Deregistering fabio
2017/04/27 09:31:25 [FATAL] ui: http: Server closed

[magiconair]

@deuch Hmm, I could check whether the SO_REUSEADDR is used. How did you kill fabio? kill -9?

Can you try with a different port or wait a bit? I think 2 min is the timeout. I just want to confirm that the vault change works.

[deuch]

I try an another port, and no need to wait, the fabio process is immediatly killed with the fatal error.

Please find the commands i use to show you the issue. I've started with the issue-274 branch and just after with the 1.4.3. You will see that the 1.4.3 works fine on the same server with the same config (i've just changed the symbolink link for the fabio binary).

-bash-4.2$ netstat -an | grep 12443
-bash-4.2$ ps -edf | grep fabio
webdev01 27842 26465  0 10:45 pts/2    00:00:00 grep --color=auto fabio
-bash-4.2$ ./fabio.start start
 Starting  Fabio
-bash-4.2$ tail -f logs/fabio.out
2017/04/27 10:45:10 [INFO] consul: Registered fabio with tags ""
2017/04/27 10:45:10 [INFO] consul: Registered fabio with health check to "http://[192.163.167.143]:9998/health"
2017/04/27 10:45:10 [INFO] HTTPS proxy listening on dckdevdck075:12443
2017/04/27 10:45:10 [INFO] Writing access log to stdout
2017/04/27 10:45:10 [INFO] Using routing strategy "rr"
2017/04/27 10:45:10 [INFO] Using route matching "prefix"
2017/04/27 10:45:10 [FATAL] listen: Fail to listen. listen tcp 192.163.167.143:12443: bind: address already in use
2017/04/27 10:45:10 [FATAL] accept tcp 192.163.167.143:12443: use of closed network connection
2017/04/27 10:45:10 [INFO] consul: Deregistering fabio
2017/04/27 10:45:10 [FATAL] ui: http: Server closed
-bash-4.2$ cd bin/
-bash-4.2$ rm fabio
-bash-4.2$ ls
fabio-1.4.2-go1.8.1-linux_amd64  fabio-1.4.3-go1.8.1-linux_amd64  fabio-custom
-bash-4.2$ ln -s fabio-1.4.3-go1.8.1-linux_amd64 fabio
-bash-4.2$ netstat -an | grep 12443
-bash-4.2$ ps -edf | grep fabio
webdev01 28201 26465  0 10:49 pts/2    00:00:00 grep --color=auto fabio
-bash-4.2$ cd ..
-bash-4.2$ ./fabio.start start
 Starting  Fabio
-bash-4.2$ tail -f logs/fabio.out
+ route add l7demo1-tomcatfabio tomcat-fabio.mydomain.com/ https://dckdevdck012:30030 opts "proto=https tlsskipverify=true"
2017/04/27 10:49:41 [INFO] consul: Registered fabio with id "l7demo1-dckdevdck075-9998"
2017/04/27 10:49:41 [INFO] consul: Registered fabio with address "192.163.167.143"
2017/04/27 10:49:41 [INFO] consul: Registered fabio with tags ""
2017/04/27 10:49:41 [INFO] consul: Registered fabio with health check to "http://[192.163.167.143]:9998/health"
2017/04/27 10:49:41 [INFO] HTTPS proxy listening on dckdevdck075:12443
2017/04/27 10:49:41 [INFO] Writing access log to stdout
2017/04/27 10:49:41 [INFO] Using routing strategy "rr"
2017/04/27 10:49:41 [INFO] Using route matching "prefix"
2017/04/27 10:49:41 [INFO] cert: Store has certificates for ["tomcat-fabio.mydomain.com,tomcat-sni.mydomain.com"]

[deuch]

Find something with the proxy address :

If i put 2 listeners in proxy.addr fabio fails to start ...

proxy.cs=cs=ssl-vault;type=vault;cert=secret/fabiodemo-001/certs;renewtoken=900s
proxy.addr = dckdevdck075:10080;proto=http,dckdevdck075:10443;proto=https;cs=ssl-vault

With this config it works with your patch (only 1 listener):

proxy.cs=cs=ssl-vault;type=vault;cert=secret/fabiodemo-001/certs;renewtoken=900s
proxy.addr = dckdevdck075:10443;proto=https;cs=ssl-vault

[magiconair]

That looks strange. I'll have a look.

[magiconair]

also fails like this

./fabio -proxy.addr ':9999,:10000'
...
2017/04/28 07:19:14 [FATAL] listen: Fail to listen. listen tcp :10000: bind: address already in use
2017/04/28 07:19:14 [FATAL] http: Server closed
2017/04/28 07:19:14 [FATAL] ui: http: Server closed

I'll open another ticket. Thanks for finding this.

[magiconair]

I've opened #279 and found the issue.

[magiconair]

I've fixed #279 and rebased the patch. Could you please try again?

[deuch]

Hi, i've made some test and i think it doesn't work as expected. In fact, the renewSelf function is called in the load function (vault_source.go) each time you want to refresh the certificates list.
In my configuration i've a lease of 20mn for a token , i set the refresh to 1m and the renewtoken to 7mn in Fabio :

proxy.cs=cs=ssl-vault;type=vault;cert=secret/fabiodemo-001/certs;renewtoken=7m;refresh=1m

But at the end, the token is renew every minutes :

{"time":"2017-05-02T15:22:10Z","type":"response","error":"","auth":{"client_token":"","accessor":"","display_name":"approle","policies":["default","fabiodemo-001/fabio-certs"],"metadata":{}},"request":{"id":"b849194f-1e95-7209-87da-3158bbba1def","operation":"update","client_token":"hmac-sha256:97a2695fb7001d40834599dee3d3b8fe3b0928747c4e026382c0662e93d04ccf","path":"auth/token/renew-self","data":{"increment":"hmac-sha256:58e823f12d627954719c9004391839f8cb6d39d9efa5269727700c04026b9ee7"},"remote_address":"192.160.120.50","wrap_ttl":0},"response":{"auth":{"client_token":"hmac-sha256:97a2695fb7001d40834599dee3d3b8fe3b0928747c4e026382c0662e93d04ccf","accessor":"hmac-sha256:ffbdbe987b69e3fde640542b39ee04a69ead65572b841a2469716e07125378e5","display_name":"approle","policies":["default","fabiodemo-001/fabio-certs"],"metadata":{}}}}
{"time":"2017-05-02T15:23:10Z","type":"response","error":"","auth":{"client_token":"","accessor":"","display_name":"approle","policies":["default","fabiodemo-001/fabio-certs"],"metadata":{}},"request":{"id":"1a9b65c7-2cde-1d6d-77a3-6c308fed0973","operation":"update","client_token":"hmac-sha256:97a2695fb7001d40834599dee3d3b8fe3b0928747c4e026382c0662e93d04ccf","path":"auth/token/renew-self","data":{"increment":"hmac-sha256:58e823f12d627954719c9004391839f8cb6d39d9efa5269727700c04026b9ee7"},"remote_address":"192.160.120.49","wrap_ttl":0},"response":{"auth":{"client_token":"hmac-sha256:97a2695fb7001d40834599dee3d3b8fe3b0928747c4e026382c0662e93d04ccf","accessor":"hmac-sha256:ffbdbe987b69e3fde640542b39ee04a69ead65572b841a2469716e07125378e5","display_name":"approle","policies":["default","fabiodemo-001/fabio-certs"],"metadata":{}}}}

[deuch]

Any update on this topic ?