How to configure TCP correctly (proxy.addr, ...)

[kopax]

Hi everyone!

Thanks for sharing Fabio, I have heard good thing about it.
I am following the project since it has been integrated in PanteraS

I wanted to try Fabio because it now has a TCP proxying mode.
I am configuring my services by registering them in consul and adding env tag to my docker containers :

urlprefix-mysite.com/                              # host specific catch all route
urlprefix-:587 proto=tcp                          # route external port 587

I wasn't able to use the tcp proxying without playing with proxy.addr in fabio.properties

1. Do I have to configure proxy.addr in fabio.properties everytime I am adding a new tcp proxied port ?
2. If yes, how can I add multiple port ? I see my configuration with a very long line in it.
3. I have consul running, isn't it possible to get this value configured automatically ?
4. I use to proxy a mail-server on HAproxy using 4 TCP ports : 25, 143, 587, 993

• I have tried to send an email from thunderbird with proto=tcp,:587;proto=tcp and I have:

  Sending of the message failed.
  An error occurred while sending mail: Unable to establish a secure link with Outgoing server (SMTP) mail.kopaxgroup.com using STARTTLS since it doesn't advertise that feature. Switch off STARTTLS for that server or contact your service provider
  

• I have tried to send an email from thunderbird with proto=tcp,:587;proto=tcp+sni it connect but I wait a very long time and I have a time out error.

I didn't had to configure anything for proxying these services before. I can see I can configure ssl keystore and tls header value. I am not sure which is the right direction. Maybe you will have any idea why it is different ?

Thanks in advance

[magiconair]

For TCP proxying to work two things need to happen:

1. fabio needs to know that traffic on a given port is to be handled by the TCP proxy (i.e. copy bytes around instead of parsing HTTP requests). This is why you have to configure the TCP listener with -proxy.addr :3306;proto=tcp. Port 3306 in this example is the port on which the exposed service is available to the outside world.

2. services which can handle that traffic need to tell fabio that they want traffic that arrives on port 3306 on fabio. The service itself runs on a random port. This is why the service advertises urlprefix-:3306 proto=tcp.

So with this in mind lets go through your questions:

1. yes, you have to configure a TCP listener on fabio for every exposed port (not service) since I haven't implemented dynamic listeners at the moment. This will come but it isn't there yet.
2. if you want to expose multiple ports do this -proxy.addr = :25;proto=tcp,:143;proto=tcp,:587;proto=tcp,:993;proto=tcp it is a comma separated list of semicolon separated k=v pairs (imagine the first entry having a default key like port)
3. not right now since this isn't a consul problem but there is no code in fabio ATM to bring up and shutdown listeners automatically. It'll come.
4. try the listener config in 2. and make sure your services are registered with the urlprefix-:25 proto=tcp, urlprefix-:143 proto=tcp ,... tags in consul

Let me know if that answers your question.

[kopax]

Hi, thanks. It get more clear for 1,2,3.

Interesting that you use mysql port :3306 for your example, is this really working ?
I was trying on panteras and haproxy to proxy postgresql and mysql database and it never worked I never used the TCP proxy for them and I never knew why.

I have tried in fabio.properties:

 proxy.addr = :25;proto=tcp,:143;proto=tcp,:587;proto=tcp,:993;proto=tcp

This work fine I can do telnet on each port. I want to send an email using port 587 with STARTTLS.
I wasn't configuring anything in HAproxy.

I have this error when I try to send an email from thunderbird:

Sending of the message failed.
An error occurred while sending mail: Unable to establish a secure link with Outgoing server (SMTP) mail.domain.com using STARTTLS since it doesn't advertise that feature. Switch off STARTTLS for that server or contact your service provider.

I am reading this but not sure if it is the right direction. Maybe it is related to proxy.header.tls proxy.header.tls.value

I haven't configured anything regarding certificate. It's always a mess to understand what needs to be configured and how it should be. Do I really have to do that here in Fabio ? I use to do SSL terminaisons on nginx. I never had to configure anything with haproxy when doing TCP proxying.

[magiconair]

proxy.header.* only matters for HTTP(S) proxies. You can terminate TLS on fabio or not. In plain TCP mode fabio should just copy the bytes it receives to the service. you can use openssl s_client for testing.

[kopax]

Thanks for your indication, apparently I have verified and SSL is fine using plain TCP, the handshake is done after Fabio.

I might have misunderstood the basic usage scenario...

I use to have HAproxy installed, and I was registering my TCP services using haproxy_tcp=587

Now I am registering with urlprefix-:587 proto=tcp AND I have to configure proxy.addr in fabio.properties

I noticed that using proxy.addr without listen address (:587;proto=tcp) will open that port on all my interfaces.

This is fine, my mail server needs to be public, but how about if I need to keep it private ? I though, ok just set the ip like (172.16.0.3:587;proto=tcp), this work, but how about if I don't know on what slave will run my application ?

Should the fabio.properties be the same for all my masters ? I was only able to access the services if I configure Fabio on that same master host (which is also a slave).

I have this assumption, if I open :587 using proxy.addr, it appear it is just for the current host application and it does not load balance anything except the local application. If this is true with what configuration can I do load balancing from a single fabio instance ?

[magiconair]

Service discovery is done through consul. Fabio finds the service instances that are registered in consul and starts routing to them once they appear and stops routing when they are gone. This is different from other proxies where you have to configure the upstream servers.

As for the bind address: proxy.addr=:587 is the same as proxy.addr=0.0.0.0:587 which will listen on all interfaces. Currently there is no method to say proxy.addr=eth0:587 but I'm planning on doing that. Therefore, you either have fabio listen on all interfaces or you have to configure it per host either through the config file or startup parameters or env vars. All will work for every parameter in fabio.properties.

[kopax]

😢 I will revert to HAproxy, but only because it's impossible to proxy using a specific interface yet and I have services that require it. For when (approx) do you schedule your release ? I am waiting for it with impatience, from what I have seen and what I have played with, it is a great tool. Very easy and handy. Thanks for sharing it.

[magiconair]

Let me see what I can hack together. This isn't difficult. Didn't have the need for it yet since you would need to run fabio via something like supervisord, daemontools or systemd anyway which can do that little bit of shell magic.

[kopax]

Sure, just tell me how I can help.

It would be nice if you could create a de configuration variable. for example, instead of using the default :3600 for 0.0.0.0:3600, use a proxy.listen_ip='172.16.0.1' variable that would be used as a default interface.
This is standard variable and it could be used in panteras with the LISTEN_IP env settings.

[magiconair]

You can always use the shell to get the ip address, e.g.:

fabio -proxy_addr "$(ifconfig eth0 | grep 'inet addr:' | awk '{print $2}' | cut -d: -f2):3600;proto=tcp"

[magiconair]

listen_ip isn't going to happen since you can have more than one listener.

[kopax]

Sorry, I meant having the possibility to configure a default interface instead of 0.0.0.0
I have set some iptables rules that configure my public interface by first allowing a few public ports then dropping the rest. It was working fine enough with HAproxy. I only had to configure one ENV tag and consul_haproxy_template was updating haproxy proxy configuration. All HAproxy exposed TCP service are accessible through the private lan interface. If I needed to open a service I only had to add a new allow rules for that new port into the iptables chain. For some reason, Fabio is exposing on top of my firewall rules because of the default 0.0.0.0, is there a way to avoid that ?

[magiconair]

As I've said, use the shell to get the ip address(es) you want fabio to bind to and then run fabio -proxy.addr "1.2.3.4:3306;proto=tcp,5.6.7.8:9876,..."

[kopax]

have this assumption, if I open :587 using proxy.addr, it appear it is just for the current host application and it does not load balance anything except the local application. If this is true with what configuration can I do load balancing from a single fabio instance ?

This is the other part that I don't understand, how do I know the ip address from the host that run Fabio ? I though load balancing was for that. If I scale up on marathon I have no way to send the new application ip to Fabio.

This is also why I do need fabio to listen by default on a different interface than 0.0.0.0 otherwise, to stop Fabio placing rules on top of mine.

[magiconair]

Maybe lets take a step back and explain what it is that you're trying to do. Since you mention marathon the setup is a bit more complex. (This should have registered when you mentioned PanteraS).

The usual deployment scenarios are listed here: https://github.com/fabiolb/fabio/wiki/Deployment

Could you explain briefly what your setup looks like?