Fabio 1.5.12 - panic: runtime error: invalid memory address or nil pointer dereference

[evandam]

After upgrading from Fabio 1.5.11, Fabio is failing to start with the following logs:

Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] Version 1.5.12 starting
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] Go runtime is go1.13.1
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] Sending metrics to StatsD on 127.0.0.1:18125 as "fabio"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] Sending metrics to StatsD on 127.0.0.1:18125 as "fabio"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] Setting GOGC=800
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] Setting GOMAXPROCS=1
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] consul: Connecting to "localhost:8500" in datacenter "kitchen"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] Admin server access mode "ro"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] Admin server listening on ":9998"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] Waiting for first routing table
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] consul: Using dynamic routes
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] consul: Using tag prefix "fabio-"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] consul: Watching KV path "/fabio/kitchen"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: 2019/11/14 22:55:19 [INFO] consul: Watching KV path "/fabio/noroute.html"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: panic: runtime error: invalid memory address or nil pointer dereference
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x4783c6]
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: goroutine 33 [running]:
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: bytes.(*Buffer).Write(0x0, 0xc0001cc000, 0xa, 0x1500, 0x1, 0x1, 0xc0002ded20)
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: #011/usr/local/Cellar/go/1.13.1/libexec/src/bytes/buffer.go:169 +0x26
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: fmt.Fprintf(0xe6a820, 0x0, 0xc5c5a4, 0xc, 0xc000499b10, 0x1, 0x1, 0x1, 0x133fa10, 0x134c040)
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: #011/usr/local/Cellar/go/1.13.1/libexec/src/fmt/print.go:205 +0xa5
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: github.com/fabiolb/fabio/route.Table.Dump(0xc00024dbc0, 0x34, 0xc000250960)
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: #011/Users/frank/src/github.com/fabiolb/fabio/route/table.go:517 +0x849
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: main.logRoutes(0xc00024dbc0, 0x0, 0x0, 0x0, 0x0, 0xc000116340, 0x6)
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: #011/Users/frank/src/github.com/fabiolb/fabio/main.go:531 +0x193
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: main.watchBackend(0xc000106600, 0xc00007cfc0)
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: #011/Users/frank/src/github.com/fabiolb/fabio/main.go:485 +0x5b6
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: created by main.main
Nov 14 22:55:19 default-ubuntu-1804 fabio[5668]: #011/Users/frank/src/github.com/fabiolb/fabio/main.go:136 +0x698
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]: 2019/11/14 22:55:19 [INFO] Setting log level to INFO
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]: 2019/11/14 22:55:19 [INFO] Runtime config
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]: {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "Proxy": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Strategy": "rnd",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Matcher": "prefix",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "NoRouteStatus": 404,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "MaxConn": 10000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "ShutdownWait": 3000000000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "DialTimeout": 3000000000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "ResponseHeaderTimeout": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "KeepAliveTimeout": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "FlushInterval": 1000000000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "GlobalFlushInterval": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "LocalIP": "10.0.2.15",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "ClientIPHeader": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "TLSHeader": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "TLSHeaderValue": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "GZIPContentTypes": null,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "RequestID": "X-Fabio-Id",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "STSHeader": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "MaxAge": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Subdomains": false,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Preload": false
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "AuthSchemes": {}
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "Registry": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Backend": "consul",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Static": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "NoRouteHTML": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Routes": ""
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "File": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "NoRouteHTMLPath": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "RoutesPath": ""
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Consul": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Addr": "localhost:8500",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Scheme": "http",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Token": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "KVPath": "/fabio/kitchen",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "NoRouteHTMLPath": "/fabio/noroute.html",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "TagPrefix": "fabio-",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Register": true,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ServiceAddr": ":9998",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ServiceName": "fabio",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ServiceTags": [
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "kitchen"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             ],
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ServiceStatus": [
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "passing"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             ],
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "CheckInterval": 1000000000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "CheckTimeout": 3000000000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "CheckScheme": "http",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "CheckTLSSkipVerify": false,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "CheckDeregisterCriticalServiceAfter": "90m",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ChecksRequired": "all",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ServiceMonitors": 1,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "TLS": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "KeyFile": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "CertFile": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "CAFile": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "CAPath": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "InsecureSkipVerify": false
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             }
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Custom": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Host": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Path": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "QueryParams": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Scheme": "https",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "CheckTLSSkipVerify": false,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "PollingInterval": 5,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "NoRouteHTML": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Timeout": 10
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Timeout": 3000000000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Retry": 500000000
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "Listen": [
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Addr": ":9999",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Proto": "http",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ReadTimeout": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "WriteTimeout": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "CertSource": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "Name": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "Type": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "CertPath": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "KeyPath": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "ClientCAPath": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "CAUpgradeCN": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "Refresh": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "Header": null,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "VaultFetchToken": ""
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "StrictMatch": false,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "TLSMinVersion": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "TLSMaxVersion": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "TLSCiphers": null,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ProxyProto": false,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ProxyHeaderTimeout": 0
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         }
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     ],
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "Log": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "AccessFormat": "$remote_host - $header.X-Forwarded-For - [$time_common] \"$request\" $response_status $response_body_size \"$header.Referer\" \"$header.User-Agent\" upstream \"$upstream_addr\"",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "AccessTarget": "stdout",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "RoutesFormat": "detail",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Level": "INFO"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "Metrics": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Target": "statsd",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Prefix": "{{clean .Exec}}",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Names": "{{clean .Host}}.{{clean .Path}}",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Interval": 10000000000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Timeout": 10000000000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Retry": 500000000,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "GraphiteAddr": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "StatsDAddr": "127.0.0.1:18125",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Circonus": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "APIKey": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "APIApp": "fabio",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "APIURL": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "CheckID": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "BrokerID": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "SubmissionURL": ""
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         }
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "UI": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Listen": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Addr": ":9998",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "Proto": "http",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ReadTimeout": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "WriteTimeout": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "CertSource": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "Name": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "Type": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "CertPath": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "KeyPath": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "ClientCAPath": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "CAUpgradeCN": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "Refresh": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "Header": null,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:                 "VaultFetchToken": ""
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "StrictMatch": false,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "TLSMinVersion": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "TLSMaxVersion": 0,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "TLSCiphers": null,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ProxyProto": false,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:             "ProxyHeaderTimeout": 0
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Color": "light-green",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Title": "kitchen ",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Access": "ro"
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "Runtime": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "GOGC": 800,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "GOMAXPROCS": 1
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "Tracing": {
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "TracingEnabled": false,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "CollectorType": "http",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "ConnectString": "http://localhost:9411/api/v1/spans",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "ServiceName": "Fabiolb",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "Topic": "Fabiolb-Kafka-Topic",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "SamplerRate": -1,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "SpanHost": "localhost:9998",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:         "TraceID128Bit": true
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     },
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "ProfileMode": "",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "ProfilePath": "/tmp",
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "Insecure": false,
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]:     "GlobMatchingDisabled": false
Nov 14 22:55:19 default-ubuntu-1804 fabio[5694]: }

[pschultz]

You can work around the issue by setting log.routes.format to something other than "detail", i.e. "delta" or "all".

[scalp42]

seeing the same issue here with 1.5.12

[evandam]

Thanks @pschultz! When is the next release scheduled that this will be included in?

[magiconair]

I’ll try to roll one this week

[dcarbone]

Thanks for getting this released so quick!