Skip to content

Refactor MultiValueHeader support#133

Closed
akdor1154 wants to merge 1 commit intoKludex:mainfrom
akdor1154:multi-value-headers
Closed

Refactor MultiValueHeader support#133
akdor1154 wants to merge 1 commit intoKludex:mainfrom
akdor1154:multi-value-headers

Conversation

@akdor1154
Copy link
Copy Markdown

@akdor1154 akdor1154 commented Jul 9, 2020

First up: sorry for dumping this on you with no prior discussion. I think it's the right solution but it is a bigger change than I initially thought would be necessary.

This PR refactors multiValueHeader support.
The main reason is ALBs: while API GW happily deals with responses with a mix of multiValueHeaders and headers, an ALB configured with a lambda backend will instead throw up a Bad Gateway error if the ALB's own multi_value_header configuration setting doesn't match what the backend returns. E.g. even the default error handler in mangum will trigger this behaviour behind an ALB configured with multi_value_headers.

My approach has thus been:

  • refactor most of test_http.py to always assume multivalueheaders is being used.
  • post-process the response["multiValueHeaders"] into response["headers"] if necessary (bool flag)
  • set the above bool flag in adaptor.py based on the presence of "multiValueHeaders" or "multiValueQueryStringParameters" in the request.

The reason for the change in approach in test_http.py is that it's easy to lossily post-process multiValueHeaders into headers, but it would be a pain to go the other way and not lose data.

I have also made some changes to adaptor.py to fix the assumption that "headers" and "queryStringParameters" will always be present (in an ALB configured with multi_value_headers, they won't be).

One side-effect of these changes is that in API GW, Mangum will now always respond with multiValueHeaders and not headers, becuase API GW always sends multiValueHeaders in its events (according to docs).

Hope this isn't too horrible to get through :)

Jarrad

Comment thread mangum/adapter.py
return logger


D = typing.TypeVar('D')
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this. I think it's fine to sprinkle # type: ignore where needed, and then revise the types later in a way that doesn't hurt readability.

Comment thread mangum/adapter.py

self.logger: logging.Logger = get_logger(self.log_level)

@staticmethod
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The behaviour of the get_header method in the HTTP protocol class is similar enough that this should be moved out into a new utils.py file and then imported in both files. The method is otherwise fine. :)

Comment thread mangum/protocols/http.py
scope: Scope
body: bytes
text_mime_types: typing.List[str]
use_multi_value_headers: bool = True
Copy link
Copy Markdown
Collaborator

@jordaneremieff jordaneremieff Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need a default? I think it should always be defined in the adapter when creating the protocol instance.

Comment thread mangum/protocols/http.py

self.response["body"] = body.decode()

if self.use_multi_value_headers == False:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if not self.use_multi_value_headers

Comment thread mangum/adapter.py
use_multi_value_headers = True
multi_value_query_string_params = event.get("multiValueQueryStringParameters")
elif event.get("queryStringParameters"):
multi_value_query_string_params = {k:[v] for k, v in event.get("queryStringParameters").items()}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it isn't a lot of effort to include test coverage this, then please add it. Otherwise it's okay to add pragma: no cover for now.

@jordaneremieff
Copy link
Copy Markdown
Collaborator

@akdor1154 thanks!

I've done just a little bit of review with some minor comments, but otherwise this seems like a good change. I'm a bit stretched for time lately, but I'll try to give it a more thorough review soon.

Well done. :)

@jordaneremieff
Copy link
Copy Markdown
Collaborator

jordaneremieff commented Jul 18, 2020

@akdor1154 I've gone through your PR, great work!

My only concern is in regards to always returning the multiValueHeaders key instead of headers in the case that someone might be doing something like:

def _handler(event, context):
    handler = Mangum(app)
    response = handler(event, context)
    headers = response["headers"]
    # do something with the headers

But I think it is okay to get this work in first, I can think more about it before the next release.

Are you able to address the comments in the review? Otherwise, I can merge this to main and make those changes myself later.

@jordaneremieff
Copy link
Copy Markdown
Collaborator

jordaneremieff commented Aug 1, 2020

@akdor1154 just a heads up, I think I will merge this and then push any adjustments separately to prepare for the next release. Please let me know if you intend to push any other changes, otherwise I'll probably tie this off when I next have time.

Thanks again! :)

@akdor1154
Copy link
Copy Markdown
Author

akdor1154 commented Aug 1, 2020 via email

@jordaneremieff
Copy link
Copy Markdown
Collaborator

@akdor1154 understandable. Thanks again, I'll go ahead and merge it in as-is when I go to prepare the next release and do the housekeeping at that point.

@jordaneremieff
Copy link
Copy Markdown
Collaborator

I think I'll close this one for now because it will end up becoming stale, and I don't think I'll be able to work on it myself (this isn't a rejection of the idea). Feel free to open a fresh PR in future for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants