Feature: Allow Wildcard Path on Host/path

[composer22]

Currently, as far as I can tell, routes need to be explicit for a given host, hence loading is very heavy because every route needs to be configured for a service.

What would be nice is a wildcard so only the host is taken into account and all traffic is redirected to that host:

For example, any of these requests:

http://fabio_server_ip:9999/foo.example.com/v1.0/about
http://fabio_server_ip:9999/foo.example.com/v1.0/login

would be redirected here:

foo.example.com/*
        foo_1                  10.0.0.2:32789
        foo_2                  10.0.0.3:32872

so foo_1 and 2 would receive this:

10.0.0.2:32789/v1.0/about
10.0.0.3:32872/v1.0/login

If there is a way to do this now, then I don't see it in the code or docs. Please explain.

I have to either be explicit or do this shit:

http://fabio_server_ip:9999/foo/v1.0/about
http://fabio_server_ip:9999/foo/v1.0/login

would be handled here:

/foo
        foo_1                  10.0.0.2:32789
        foo_2                  10.0.0.3:32872

and foo_1 and 2 receive this:

10.0.0.2:32789/foo/v1.0/about
10.0.0.3:32872/foo/v1.0/login

so now the server has to account for the hostname in the path which is BS.

[magiconair]

Matching happens on the longest path. So what you're looking for is for your service to either announce / or /foo/v1.0. You can have more specific routes later. If you have another service that handles /foo/v1.0/user then that will take precendence.

You can also match on foo.example.com/ or foo.example.com/foo/v1.0/

[composer22]

Thanks for your patience. I'm not sure I understand fully, but I will try and qualify my question.

I'd like all paths to fall through to the targets based only on the hostname. To do so, I want to only have to do this:

SERVICE_TAGS=urlprefix-lb-hello-world.example.com/

or this:

SERVICE_TAGS=urlprefix-lb-hello-world.example.com/*

and no matter what i send as a path, as long as the hostname matches, it will send to one of the assigned targets in the table for lb-hello-world.example.com/ or lb-hello-world.example.com/*.

If, for example, lb-hello-world.example.com was in the DNS as pointing to fabio's IP address, lets as 192.168.1.4.

(and I'm also using fabio for multiple lb endpoints to other services):

/etc/hosts:

192.168.1.4    fabio          
192.168.1.4    lb-hello-world.example.com
192.168.1.4    lb-checkout.example.com

These requests for lb-hello-world.example.com => favio...

http://lb-hello-world.example.com:9999/v1.0/hello
http://lb-hello-world.example.com:9999/v.10/health
http://lb-hello-world.example.com:9999/

...would all resolve to the following single rule:

lb-hello-world.example.com/*
        hello_world_1       10.0.0.2:32789
        hello_world_2       10.0.0.3:32872      

...and hello_world_1 or hello_world_2 services would receive these requests as:

http://10.0.0.2:32789/v1.0/hello
http://10.0.0.3: 32872/v.10/health
http://10.0.0.2:32789/

This is not happening now. Here is what is happening...

From my hosts file (scrubbed):

52.35.xxx.xxx   lb-hello-world.example.com

From my fabio logs using curl testing:

2016/02/03 18:37:36 [WARN] No route for lb-hello-world.example.com:9999/v1.0/health
2016/02/03 18:37:51 [WARN] No route for lb-hello-world.example.com:9999/v1.0/health/
2016/02/03 18:38:10 [WARN] No route for lb-hello-world.example.com:9999/v1.0/hello
2016/02/03 18:38:18 [WARN] No route for lb-hello-world.example.com:9999/v1.0/hello/

From my routing table:

#   Service         Host                   Path  Dest                        Weight
1   hello_world_1   lb-hello-world.example.com   /  http://10.0.77.64:32772/    100%

I do not want to pass on the hostname as apart of the path for example:

Not this; no host:

#   Service         Host                   Path  Dest                        Weight
1   hello_world_1            /lb-hello-world.example.com    http://10.0.77.64:32772/    100%

...or this:

#   Service         Host                   Path  Dest                        Weight
1   hello_world_1            /hello-world   http://10.0.77.64:32772/    100%

or send fabio this:

http://fabio:9999/hello-world/v1.0/hello
http://fabio:9999/hello-world/v.10/health
http://fabio:9999/hello-world

as hello_world_1 and hello_world_2 are receiving this...

http://10.0.0.2:32789/hello-world/v1.0/hello
http://10.0.0.3: 32872/hello-world/v.10/health
http://10.0.0.2:32789/hello-world

That is what is happening now.

Again I don't want to define every path that a server/service is providing. There might require hundreds of entries per service.
Again I don't want to parse out the higher level path in the service as a work around. e.g defining routes in the application with /hello-world/v.10/health

NO:
mux.HandleFunc("/lb-hello-world.example.com/v.10/health", s.helloHandler)
or
mux.HandleFunc("/hello-world/v.10/health", s.helloHandler)

YES:
mux.HandleFunc("/v.10/health", s.helloHandler)

[magiconair]

If you provide SERVICE_TAGS=urlprefix-foo.com/ for every instance then you get the desired behavior.

That assumes that every such instance can handle every request for foo.com. As long as that is true you're good.

[magiconair]

Also, the best is just to try it and provide a specific example of what doesn't work as expected.

[composer22]

Ok, let me try some examples and report back my results. bbiab

[composer22]

Feedback:

Here is my test...

hosts file changes (scrupped):

$ cat /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1   localhost
255.255.255.255 broadcasthost
::1             localhost 

52.35.xxx.xxx   lb-hello-world.example.com

docker command to start service end point:

docker run -d \
  -P \
  -e constraint:com.military.role==worker  \
  -e "SERVICE_NAME=hello_world_1"  \
  -e "SERVICE_CHECK_HTTP=/v1.0/health"  \
  -e "SERVICE_CHECK_INTERVAL=5s" \
  -e "SERVICE_CHECK_TIMEOUT=15s"  \
  -e "SERVICE_TAGS=urlprefix-lb-hello-world.example.com/"  \
  --restart=always  \
  --name hello_world_1  \
  composer22/hello-world

Registers with fabio. Yippie!:

Curl examples:

curl -v http://lb-hello-world.example.com:9999/v1.0/health/
curl -v http://lb-hello-world.example.com:9999/v1.0/health
curl -v http://lb-hello-world.example.com:9999/v1.0/hello/
curl -v http://lb-hello-world.example.com:9999/v1.0/hello

Fabio Log Results:

2016/02/03 20:42:22 [WARN] No route for lb-hello-world.example.com:9999/v1.0/health
2016/02/03 20:42:25 [WARN] No route for lb-hello-world.example.com:9999/v1.0/health/
2016/02/03 20:42:31 [WARN] No route for lb-hello-world.example.com:9999/v1.0/hello
2016/02/03 20:42:33 [WARN] No route for lb-hello-world.example.com:9999/v1.0/hello/

Command line partial example results:

$ curl -v http://lb-hello-world.example.com:9999/v1.0/health
*   Trying 52.35.xxx.xxx...
* Connected to lb-hello-world.example.com (52.35.xxx.xxx) port 9999 (#0)
> GET /v1.0/health HTTP/1.1
> Host: lb-hello-world.example.com:9999
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Date: Wed, 03 Feb 2016 20:46:06 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host lb-hello-world.example.com left intact
[12:46:06]
$ curl -v http://lb-hello-world.example.com:9999/v1.0/health/
*   Trying 52.35.xxx.xxx...
* Connected to lb-hello-world.example.com (52.35.xxx.xxx) port 9999 (#0)
> GET /v1.0/health/ HTTP/1.1
> Host: lb-hello-world.example.com:9999
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Date: Wed, 03 Feb 2016 20:46:09 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host lb-hello-world.example.com left intact
[12:46:09] 
$ 

I know the endpoint is good.
I know it never reached the endpoint because a) no log entry b) it would return a UUID in the response header.

[magiconair]

I think I get it. Try this: curl -v -H 'Host: lb-hello-world.example.com' 'http://lb-hello-world.example.com:9999/v1.0/hello. fabio uses the Host header to match the request with the routing table. Since your request has a Host header of lb-hello-world.example.com:9999 and not lb-hello-world.example.com there is no match. Hence the 404.

[magiconair]

Also, see this: https://github.com/eBay/fabio#debugging

[magiconair]

And https://github.com/eBay/fabio#request-tracing might come in handy, too.

[composer22]

Bingo! Beautiful!

VERY VERY interesting.

$ curl -v -H "Host: lb-hello-world.example.com" "http://lb-hello-world.example.com:9999/v1.0/hello"
*   Trying 52.35.xxx.xxx...
* Connected to lb-hello-world.example.com (52.35.xxx.xxx) port 9999 (#0)
> GET /v1.0/hello HTTP/1.1
> Host: lb-hello-world.example.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Length: 12
< Content-Type: application/json;charset=utf-8
< Date: Wed, 03 Feb 2016 20:57:35 +0000
< X-Request-Id: BC46FB26-8E0C-47AF-BD5E-B3DA729B564E
< 
* Connection #0 to host lb-hello-world.example.com left intact

This might be a challenge then. I guess if I have a Proxy front end, it will have to set the host header in the RedirectRule before calling fabio.example.com:9999. This unless because I am testing locally with a hacked host file, it isn't propagated correctly.

Thanks again for your help. √ 👍 💯

[composer22]

Didnt see in docs so perhaps an update there.

thanks again

[magiconair]

No problem. I'm glad we found it. What did you not see in the docs? The links I've posted are referenced from https://github.com/eBay/fabio#documentation albeit at the bottom of the page.

The way you probably deploy fabio is either behind a load balancer which terminates SSL and distributes traffic to multiple fabio instances or you run fabio on port 80/443 on the internet directly. In both cases the Host header will be set correctly since reverse proxies shouldn't mess with the headers from the incoming request.

Maybe this helps: https://github.com/eBay/fabio#deployment

Let me know what else is unclear.

[composer22]

It's deeply embedded in the docs under testing. I see this now. I was trying to follow the examples in the context of automated service discovery with Consul. There wasn't anything here which demonstrated hostname/ default with the idea of fabio acting as a general router/lb to many different hostnames.

Where we are headed is rather simple and somewhat based on what we have now in prod.

Internet traffic being directed to first layer proxy servers with redirect. These servers direct traffic with complex set of rules to a fabio layer.

The fabio layer lbs directs main site or subservices.

The fabio layer might be split:

3 fabios highly available each handling:

• www.foo.com
• mobile.foo.com
• news.foo.com

another 3 fabios each handling:

• auth.foo.com
• checkout.foo.com
• etc.

and so on.

The subservices also access the fabio instances directly.

The question for me which I don't know is if, for example, I created a rewrite rule directing traffic away from the main site to a subdomain/service, whether it will carry forward the new HOSTNAME
ex:

DNS is pointing to same...

192.168.2.3        lb1-fabio.foo.com
192.168.2.3        lb1-news.foo.com

proxy rule in Apache, nginx, or HAProxy maps via Virtual Load Balancer to three Fabio instances in RR.

www.foo.com/latest/$1  ==>   
     lb1-news.foo.com:9999/latest/$1
     lb2-news.foo.com:9999/latest/$1
     lb3-news.foo.com:9999/latest/$1

Say lb1-fabio.foo.com is chosen...

The routes in lb1-fabio.foo.com:

lb1-news.foo.com/
       news_1         10.0.0.2
       news_2         10.0.0.3
 lb1-mobile.foo.com/
       mobile_1         10.0.0.4
       mobile_2         10.0.0.5

Q: Will HOSTNAME be set appropriately in the proxy redirected call to fabio http://lb1-news.foo.com:9999?

I need to explore this...

[magiconair]

To be honest your setup sounds rather complex and I think you're missing the point of the design goal of fabio a bit. The way we are running fabio is as follows:

Internet -> LB (term SSL) -> N x fabio -> services

The LB has only a single rule: route all traffic to fabio.

The main motivation for developing fabio is simplification: to eliminate the complex routing and rewriting logic that we had to replicate across the LBs and that we had to keep in sync with the services we've deployed. So the LB config became essentially a vital part of the service configuration without the service would not function but it was managed differently (puppet vs. git)

Since fabio builds the routing configuration on-the-fly from an up-to-date service registry the inbound LB becomes a simple SSL terminator or you run fabio directly on the internet as your front-facing service and get rid of the LB altogether.

Also, depending on your traffic you may not need several fabio clusters. We're running comfortably 5-10k req/sec over a group of fabio servers and that can be scaled up easily by just adding more instances. So instead of having two groups of 3 fabios you might be fine with just one group of 3 and add more instances as you see fit.

Last but not least we use fabio only for the inbound routing, i.e. directing internet traffic to the services but not for communication among the services themselves since you use consul with service registration and service discovery for this.