websockets failing with 500 response - running rancher behind fabio

[jamessammut]

Running fabio:1.2-go1.6.3.

I have run this successfully <1.2 but recently upgraded to get the https support.

When I run a direct connection from chrome to rancher it works so I can confirm rancher is running properly. Running rancher:v1.1.2

Copy of the request headers (fails with 500 error response):

GET ws://rancher.cloud.XXXX.com/v1/projects/1a16602/subscribe?eventNames=resource.change&eventNames=service.kubernetes.change&limit=-1&include=hosts&include=services&include=instances&include=instance&include=instanceLinks&include=ipAddresses&sockId=36 HTTP/1.1
Host: rancher.cloud.XXXX.com
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
Origin: http://rancher.cloud.XXXX.com
Sec-WebSocket-Version: 13
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: PL=rancher; token=TcJiLGPNKqYePcAC4ZNYTTSApPyV5bYr1ujqAAbG; CSRF=A67819676337CDF4C6EF3591C68B40B4
Sec-WebSocket-Key: PzitR2II+lS9zhe9BtGo7w==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

[magiconair]

@jamessammut Is there anything in the logs of either fabio or rancher? I've just tested fabio with websocket support with the included demo/server -prefix /echo -proto ws and demo/wsclient and that works just fine.

[jamessammut]

What is the best option to run fabio with verbose logs?

I see this error in the Rancher logs, so I guess fabio is routing this request to Rancher:

2016-07-25 14:38:42,545 ERROR [:] [] [] [] [1976473931-1222] [i.g.i.g.s.ApiRequestFilterDelegate  ] Unhandled exception in API for request [ws://rancher.cloud.XXXX.com/v1/projects/1a16602/subscribe] io.github.ibuildthecloud.gdapi.exception.ClientVisibleException: null

I might also raise something on their forums.

[magiconair]

@jamessammut there is no such thing. Might be worth adding. The question is if it works with an older version but not with the most recent version then what has changed? Can you capture the requests with tcpdump and/or WireShark before and after fabio? Maybe fabio 1.2 is sending something different that makes rancher behave that way.

[magiconair]

@jamessammut I am going to close this ticket for now until there is more evidence pointing towards fabio. In that case please comment or re-open (not sure if you can but I will).

[jamessammut]

I've taken Fabio out of the path and Rancher works fine. I will attempt to
gather some info for a possible trouble shoot when I get the chance.

[magiconair]

@jamessammut Best thing would be to collect some tcpdump logs from both sides of the fabio connection. That might tell something.

[magiconair]

Closing again since there is no feedback for 4 weeks. Feel free to re-open when new data arrives.

[aaronhurt]

@jamessammut Did you ever find a solution to this? We're seeing something similar with the latest 1.4.3 fabio release and rancher 1.6.0

[jamessammut]

@juliamgamble Julian is this still a problem? I don't work on that project currently but Julian is still involved. Julian do you still see this issue?

[ketchoop]

I've got the same thing with rancher. And, maybe, can collect some needed data that is necessary to work it out. @magiconair What is needed? Tcpdump, access logs, response and request headers?