How to load balance '/'?

[stevej]

Hello!

I've been trying to evaluate fabio as a reverse proxy but am having trouble configuring it for our use case: to load balance all traffic (specifically / in this case) to multiple backends. I've gone over the documentation a number of times and spent about half a day trying to configure it for this case with no luck and was hoping you could provide some guidance.

I'm running consul in dev mode with./consul agent -dev -config-dir consul.d with the contents of consul.d/web.json being three test backends all serving a static document at / (Please ignore the silly checks for now)

stevej@proxy-test-16:~/consul.d$ ls
web.json
stevej@proxy-test-16:~/consul.d$ cat web.json
{
  "services": [
    {
      "name": "web",
      "id": "perf-target-1",
      "tags": ["web"],
      "address": "10.240.0.6",
      "port": 4140,
      "checks":[{"script": "/bin/true"}]
    },
    {
      "name": "web",
      "tags": ["web"],
      "id": "perf-target-2",
      "address": "10.240.0.5",
      "port": 4140,
      "checks":[{"script": "/bin/true"}]
    },
    {
      "name": "web",
      "tags": ["web"],
      "id": "perf-target-3",
      "address": "10.240.0.7",
      "port": 4140,
      "checks": [{"script": "/bin/true"}]
    }
  ]
}

I'm running fabio using the instructions in the README and running as bin/fabio with no properties.

When I send a request to / or /web I receive a 404.

stevej@proxy-test-16:~/go$ curl -vvvv http://localhost:9999/web
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9999 (#0)
> GET /web HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:9999
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Fri, 15 Apr 2016 16:37:36 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact

I've confirmed that fabio is bound to that port with netstat -anp

tcp6       0      0 :::9999                 :::*                    LISTEN      3815/fabio

Thanks in advance for any help.

[magiconair]

Have a look at the Quickstart: https://github.com/eBay/fabio#quickstart

You need to add a urlprefix-/ tag to all service definitions if they handle /. The idea behind fabio is that services announce which endpoints they handle during service registration and they do that by adding a urlprefix-host/path tag for every endpoint they accept.

[magiconair]

Also, the checks are vital since fabio will only add services with a passing health check to the routing table. So it is good that you have them. :) Another common mistake are non-unique service IDs for different instances of the same service.

[stevej]

Hi, thanks for the reply!

I went ahead and added urlprefix-/ tags but am still getting a 404 and an error No route for web/ from fabio's stdout. Are my tags still malformed?

$ cat consul.d/web.json
{
  "services": [
    {
      "name": "web",
      "id": "perf-target-1",
      "tags": ["urlprefix-web-/"],
      "address": "10.240.0.6",
      "port": 4140,
      "checks":[{"script": "/bin/true"}]
    },
    {
      "name": "web",
      "tags": ["urlprefix-web-/"],
      "id": "perf-target-2",
      "address": "10.240.0.5",
      "port": 4140,
      "checks":[{"script": "/bin/true"}]
    },
    {
      "name": "web",
      "tags": ["urlprefix-web-/"],
      "id": "perf-target-3",
      "address": "10.240.0.7",
      "port": 4140,
      "checks": [{"script": "/bin/true"}]
    }
  ]
}
stevej@proxy-test-16:~$ curl -H "Host: web" -vvv localhost:9999
* Rebuilt URL to: localhost:9999/
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9999 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Accept: */*
> Host: web
>
< HTTP/1.1 404 Not Found
< Date: Fri, 15 Apr 2016 17:18:26 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact

[magiconair]

urlprefix-/

[stevej]

Unfortunately, I get the same 404 with urlprefix-/

Here is some output from fabio's stdout in case it's useful.

2016/04/15 19:01:21 [INFO] Setting GOGC=800
2016/04/15 19:01:21 [INFO] Setting GOMAXPROCS=16
2016/04/15 19:01:21 [INFO] Metrics disabled
2016/04/15 19:01:21 [INFO] consul: Connecting to "localhost:8500" in datacenter "dc1"
2016/04/15 19:01:21 [INFO] consul: Registered fabio as "fabio-proxy-test-16-9998" with address ":9998" and health check to "http://[10.240.0.9]:9998/health"
2016/04/15 19:01:21 [INFO] Admin server listening on ":9998"
2016/04/15 19:01:21 [INFO] consul: Using dynamic routes
2016/04/15 19:01:21 [INFO] consul: Using tag prefix "urlprefix-"
2016/04/15 19:01:21 [INFO] consul: Watching KV path "/fabio/config"
2016/04/15 19:01:21 [INFO] Using routing strategy "rnd"
2016/04/15 19:01:21 [INFO] HTTP proxy listening on :9999
2016/04/15 19:01:21 [INFO] consul: Health changed to #8
2016/04/15 19:01:21 [INFO] consul: Manual config changed to #1
2016/04/15 19:01:21 [INFO] Updated config to
2016/04/15 19:01:21 [INFO] consul: Health changed to #9
2016/04/15 19:01:25 [WARN] No route for web/

[magiconair]

I don't see a route add coomand in the log. Therefore, your service is not included. Can you replace your health check with an http check instead of the script? Did you reload consul after the change? Does consul show the service green ?

[magiconair]

What does curl localhost:8500/v1/health/state/any?pretty say?

[stevej]

Switching to http-based health checks brought the cluster up. Sorry for the consul noobishness, it's not something I use on a regular basis.

[stevej]

I have this working now so I'm going to close this ticket. The numbers i'm seeing are better than advertised: 28k qps using 6 cores on a 16-core GCE vm instance using about 45mb of RAM. p90 was 1ms, p95 was 5, p99 was 600.

Thank you for all your help!

[magiconair]

No problem. Glad you got it working. Let me know if there are other issues.