Using upstream hostname for request

[chad-codes]

My team and I are working on figuring out a service discovery architecture for our multitude of Serverless stacks.

I've created two EC2 instances in AWS: one with Consul and the other with Fabio. I have a Serverless stack deployed with a single endpoint: /status. I registered a service in Consul with the address being the API Gateway URL (xxxxxxx.execute-api....com) and the port to 443. I set the tag to urlprefix-/status proto=https.

Now, maybe I'm confused on how all of this works, but how come I can't do curl <fabio-ip>:9999/status? When I run that command, I get a 403 status code and a body saying bad request from CloudFront. Adding a Trace header prints Routing to service status on https://xxxxxxx.execute-api....com:443/status. If I run curl https://xxxxxxx.execute-api....com:443/status, I get back a response from my lambda function as I expect.

It seems in order for it work, I need to add a Host header matching the API Gateway URL: curl -H 'Host: xxxxxxx.execute-api....com' <fabio-ip>:9999/status. That works fine and returns back a response from my lambda function, but that defeats the whole purpose. If I have clients go to curl the api, I can't expect them to know the API Gateway URL since Fabio and Consul are meant to abstract it.

So, my main question is, what am I doing wrong? Am I thinking about this in the wrong way? Am I missing some configuration of some sort? Thanks.

[magiconair]

That should work. You're sure that your urlprefix- tag does not contain the hostname?

[mitchelldavis]

To piggy back on @CGreenburg question. We're hoping to get Fabio to work like a reverse proxy (I think). HTTP -> HTTPS in this case so our users can simply hit Fabio and Fabio routes the traffic to the backend services. (An obvious use case for the tool.) However, we're stubbing our toes somewhere.

http://<fabio machine>/<service name>/<path> --> https://<service address>/<path>

• I think I understand that a straight through proxy wouldn't work in this case because the traffic would need to be terminated in Fabio and then continued on to the service and back in order to handle the certificates.
• We were able to get this to work if we added the Host: <service address> to the request which is prohibitive. If the software knew the host header then it could go strait there.

What are we missing? Are we fundamentally missing something on how this should all work, or missing something in the Fabio configuration?

[mitchelldavis]

Here is my configuration for a Serverless REST API in Fabio:

/registrar	https://<garble>.amazonaws.com:443	proto=https tlsskipverify=true	100.00%

(purposefully garbling the url there.)

I get a Bad Request from Amazon when I run even though this is the use case we're looking for...

curl http://<ip>:9999/registrar

I should also note that I'm using the latest version of the magiconair/fabio docker container to run Fabio.

[magiconair]

@mitchelldavis Pls open another ticket in the future. I don't mind answering several questions but I'd prefer not to morph or merge issues. You probably need to add strip=/servicename to the urlprefix tag since fabio doesn't do automatic path stripping.

[chad-codes]

@magiconair yes, the tag does not contain the hostname. I've noticed when running curl -v <fabio-ip>:9999/status, the Host header gets set to <fabio-ip>:9999. If I set the Host header to only the ip, I get the same result.

I registered a service pointing to www.google.com port 443 with a tag urlprefix-/google proto=https strip=/google. When I run curl -v -L <fabio-ip>:9999/google, the Host gets set to <fabio-ip>:9999 but I don't get the html from google. If I run curl -v -L -H 'Host: <fabio-ip>' http://<fabio-ip>:9999/google without the port specified in the host header, everything works like a charm.

So.... I'm unsure as to why the API Gateway URL is so picky about the Host header...

[magiconair]

That looks like a bug.

[mitchelldavis]

@magiconair Sorry to confuse. @CGreenburg sits behind me, and we're working through the same problem.

[magiconair]

Indeed - that should have been obvious. Sorry for pushing :)

[magiconair]

What does your consul service record look like?

[mitchelldavis]

[
    {
        "Node": "api",
        "Address": "opesqzhtvl.execute-api.us-west-2.amazonaws.com",
        "TaggedAddresses": null,
        "ServiceID": "registrar",
        "ServiceName": "registrar",
        "ServiceTags": [
            "urlprefix-/registrar proto=https strip=/registrar"
        ],
        "ServiceAddress": "",
        "ServicePort": 443,
        "ServiceEnableTagOverride": false,
        "CreateIndex": 517,
        "ModifyIndex": 517
    }
]

[magiconair]

Can you try setting the ServiceAddress ?

[mitchelldavis]

I did and same result:

[
    {
        "Node": "api",
        "Address": "opesqzhtvl.execute-api.us-west-2.amazonaws.com",
        "TaggedAddresses": null,
        "ServiceID": "registrar",
        "ServiceName": "registrar",
        "ServiceTags": [
            "urlprefix-/registrar proto=https strip=/registrar"
        ],
        "ServiceAddress": "opesqzhtvl.execute-api.us-west-2.amazonaws.com",
        "ServicePort": 443,
        "ServiceEnableTagOverride": false,
        "CreateIndex": 745,
        "ModifyIndex": 745
    }
]

[magiconair]

And the service is accessible under https://opesqzhtvl.execute-api.us-west-2.amazonaws.com/ ?

[mitchelldavis]

Yes. We can curl it directly no problem.

[mitchelldavis]

you should be able to curl it at: https://opesqzhtvl.execute-api.us-west-2.amazonaws.com/dev