runtime error: integer divide by zero

[bodiug]

Hi,

I've run into a bug. When running a service consisting of around 100 containers this error does not occur, but scaling up to several 100's it does. Other services with different urlprefix still continue to work.

Nov 15 09:47:58 random-host fabio[471]: route add local-bla-app hello-world.domain.com/ http://192.168.10.55:8080/ tags "random-host-urlprefix-hello-world.domain.com/"
Nov 15 09:48:37 random-host fabio[471]: 2016/11/15 09:48:37 http: panic serving [::1]:46124: runtime error: integer divide by zero
Nov 15 09:48:37 random-host fabio[471]: goroutine 28 [running]:
Nov 15 09:48:37 random-host fabio[471]: net/http.(*conn).serve.func1(0xc421e0c400)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/go1.7.3/src/net/http/server.go:1491 +0x12a
Nov 15 09:48:37 random-host fabio[471]: panic(0x7d13c0, 0xc42000e070)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/go1.7.3/src/runtime/panic.go:458 +0x243
Nov 15 09:48:37 random-host fabio[471]: github.com/eBay/fabio/route.glob..func1(0x0, 0xc421d44101)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/gopath/src/github.com/eBay/fabio/route/picker.go:48 +0x92
Nov 15 09:48:37 random-host fabio[471]: github.com/eBay/fabio/route.rndPicker(0xc42004f680, 0x1)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/gopath/src/github.com/eBay/fabio/route/picker.go:30 +0x36
Nov 15 09:48:37 random-host fabio[471]: github.com/eBay/fabio/route.Table.lookup(0xc421d420c0, 0xc421e102b0, 0xf, 0xc421e10294, 0x1, 0x0, 0x0, 0x6563617254)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/gopath/src/github.com/eBay/fabio/route/table.go:281 +0x3ea
Nov 15 09:48:37 random-host fabio[471]: github.com/eBay/fabio/route.Table.Lookup(0xc421d420c0, 0xc4200b33b0, 0x0, 0x0, 0x0)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/gopath/src/github.com/eBay/fabio/route/table.go:253 +0x226
Nov 15 09:48:37 random-host fabio[471]: github.com/eBay/fabio/proxy.target(0xc4200b33b0, 0x7f5f2453b401)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/gopath/src/github.com/eBay/fabio/proxy/util.go:94 +0x8b
Nov 15 09:48:37 random-host fabio[471]: github.com/eBay/fabio/proxy.(*httpProxy).ServeHTTP(0xc42019e0f0, 0xa1d900, 0xc420165040, 0xc4200b33b0)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/gopath/src/github.com/eBay/fabio/proxy/proxy.go:35 +0x9c
Nov 15 09:48:37 random-host fabio[471]: net/http.serverHandler.ServeHTTP(0xc420070100, 0xa1d900, 0xc420165040, 0xc4200b33b0)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/go1.7.3/src/net/http/server.go:2202 +0x7d
Nov 15 09:48:37 random-host fabio[471]: net/http.(*conn).serve(0xc421e0c400, 0xa1e100, 0xc420c847c0)
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/go1.7.3/src/net/http/server.go:1579 +0x4b7
Nov 15 09:48:37 random-host fabio[471]: created by net/http.(*Server).Serve
Nov 15 09:48:37 random-host fabio[471]: /Users/frschroeder/go1.7.3/src/net/http/server.go:2293 +0x44d
Nov 15 09:48:57 random-host fabio[471]: 2016/11/15 09:48:57 [INFO] consul: Health changed to #107221
Nov 15 09:49:35 random-host fabio[471]: 2016/11/15 09:49:35 [INFO] consul: Health changed to #107231
Nov 15 09:49:42 random-host fabio[471]: 2016/11/15 09:49:42 [INFO] consul: Health changed to #107233
Nov 15 09:49:44 random-host fabio[471]: 2016/11/15 09:49:44 [INFO] consul: Health changed to #107235
Nov 15 09:49:44 random-host fabio[471]: 2016/11/15 09:49:44 [INFO] consul: Health changed to #107396
Nov 15 09:49:44 random-host fabio[471]: 2016/11/15 09:49:44 [INFO] Updated config to
Nov 15 09:49:44 random-host fabio[471]: route add local-bla-app hello-world.domain.com/ http://192.168.8.8:8080/ tags "random-host-urlprefix-hello-world.domain.com/"

[magiconair]

On vacation right now but I'll have a look later. That looks like an edge case I've missed.

[bodiug]

No worries, thanks for the heads-up. Enjoy the remaining of your vacation!

[magiconair]

You've indeed found a bug and it is in code for which I don't even have a test... How embarrassing.

What happens is that the weighing algorithm which pre-computes the distribution of the target URLs based on weight allocates only 100 slots (= 100%) and when you create more than a couple hundred instances the rounding errors ensure that no instance is used. This then trips the random number function which never thought it would be called with zero. Why would anybody create more than 100 instances anyway? ;)

Time to refactor. Not a biggie but I need to add a test first. For now stay <= 100 instances per prefix.

I've got an ugly patch which you can apply if this is an urgent production issue. Otherwise, a fix is coming soon.

Thanks for finding this.

[bodiug]

Thanks for diving into this, no ugly patching needed. :) This was found during testing 100+ nodes. But don't hesitate to ping me if I can help testing.

[killcity]

Thanks for posting this bug :) I'm going to be rolling out a single urlprefix to more than 100 nodes, relatively soon (production).

[killcity]

@magiconair Any idea when this might be resolved? I know you've been all over the place :) We will be rolling out to a couple hundred nodes sometime in Jan/Feb timeframe.

Cheers

[magiconair]

One of the next things on my list.

[magiconair]

@killcity I'll have a look today whether there is a quick workaround. My guess is that changing 100 to 1000 or 10000 in https://github.com/eBay/fabio/blob/master/route/route.go#L222 would do the trick but I need to write a small test to verify. If that works then I could push this first and then deliver a real fix later.

[magiconair]

@killcity Unfortunately, the really quick fix doesn't work. On the upside, I've found that I actually do have a test for the code. Just in a place where I didn't expect it :) I'll keep looking.

[killcity]

@magiconair Thanks!