Mailgun: Separate configuration of "Private API key" vs. "HTTP webhook signing key" [after Mailgun key rotation]

[dominik-lekse]

Description

• I currently evaluate Anymail in my Django project to integrate with Mailgun to be able to process inbound mails via Mailgun routes and the webhook of Anymail and to send mails via Mailgun.
• The Mailgun API security distinguishes between a "Private API key" and a "HTTP webhook signing key" which are different.
• According to the Anymail documentation, the Mailgun ESP needs to be provided the ANYMAIL_MAILGUN_API_KEY setting which takes the Mailgun "Private API key".
• However, if ANYMAIL_MAILGUN_API_KEY is the "Private API key", the verification of inbound mail via the webhook fails
• Webhook HTTP requests from Mailgun are signed with the "HTTP webhook signing key"

Proposal

• Define a new setting ANYMAIL_MAILGUN_SIGNING_KEY which must be configured with the "HTTP webhook signing key" when using the inbound mail functionality
• MailgunBaseWebhookView should take the key to verify the signature from ANYMAIL_MAILGUN_SIGNING_KEY

(Kind of) workaround

• Depending on which API key is provided to ANYMAIL_MAILGUN_API_KEY, either inbound mails from Mailgun ESP or send mails via Mailgun ESP works, but not both
• To allow inbound mails, set ANYMAIL_MAILGUN_API_KEY to the "HTTP webhook signing key"
• To allow sending mails, set ANYMAIL_MAILGUN_API_KEY to the "Private API key"

Some notes

• Before opening this issue, I have carefully scanned the Anymail documentation but was not able to find a hint
• I have rotated all Mailgun keys at least once

Environment

• Python 3.7.3
• Anymail version: 6.0.1
• ESP: Mailgun

[mbk-ok]

Hi Dominik, the webhook signing key and API key are two different keys. They are not the same nor are they used for any of the same purposes. Do you want to submit a support ticket to help troubleshoot? Reach out to help@mailgun.com.

[medmunds]

@mbk-ok Anymail maintainer here. (I'm also having a conversation with Conor in your site chat righ tnow.)

All of Mailgun's documentation currently describes using "your Mailgun API key" as the key to calculate webhook validation -- and this is how Anymail has always done it. Is the webhook signing key (as separate from the API key) a recent change, and not yet reflected in the docs?

[mbk-ok]

Upon instantiation, I believe you're right, that the API key and webhook signing key are the same. However, if either one is purged/rotated, then they become different. Does that help clarify?

[medmunds]

Hmm... that makes sense. So developers will only run into this problem after rotating one of their keys. Is there a way @dominik-lekse can get both API key and webhook signing key matching again, to avoid needing to wait for an Anymail patch?

It would be helpful to clarify the different keys in Mailgun's documentation (and perhaps initialize the webhook signing key to differ from the API key so this behavior will be immediately apparent to developers). Every docs reference I could find -- and my own testing (since I hadn't rotated keys) -- says to use the Mailgun API key for webhook validation.

Here are three misleading docs references:

• https://documentation.mailgun.com/en/latest/user_manual.html#webhooks "Encode the resulting string with the HMAC algorithm (using your API Key as a key and SHA256 digest mode)."
• https://www.mailgun.com/guides/your-guide-to-webhooks "... encode the resulting string with the HMAC algorithm (using the ESP’s supplied API Key as a key and SHA256 digest mode)"
• https://www.mailgun.com/blog/a-guide-to-using-mailguns-webhooks "Encode the resulting string with HMAC, using your Mailgun API key as the key and Sha256 as the algorithm."

[Also, @dominik-lekse thanks for the detailed problem report.]

[mbk-ok]

As for getting the two keys to match for @dominik-lekse, I certainly don't have that power, nor do I believe even our Support or Development teams have that ability. He could submit a support ticket to give it a try, but I'm skeptical.

I can create a PR to reflect these differences in the keys in our documentation, or you can if you like.

[medmunds]

@mbk-ok big thanks for jumping in quickly with the accurate explanation—I would have spent a long time trying to figure out why my test environment was behaving differently. I'll let you update the Mailgun docs. (I don't think your docs repository includes all of the references above.)

@dominik-lekse yes, we'll want to implement your proposal to allow separate Mailgun API key and webhook signing key in the Anymail settings. Let me know if you'd like to take a crack at a PR.

I'd suggest the new setting should be (ANYMAIL_)MAILGUN_WEBHOOK_SIGNING_KEY (include the word WEBHOOK). And for backwards compatibility it should fall back to the MAILGUN_API_KEY if a separate webhook signing key isn't provided.

Also, here's a workaround for current Anymail releases that lets you both send messages and receive webhook events. It uses Django's urlconf arguments to class-based views to override the API key only for Anymail's Mailgun webhook views, while leaving the original API key in place for send API calls.

In your project's urls.py...

# Add this:
from django.conf import settings
from anymail.webhooks.mailgun import MailgunInboundWebhookView, MailgunTrackingWebhookView

MAILGUN_WEBHOOK_SIGNING_KEY = settings.ANYMAIL["MAILGUN_WEBHOOK_SIGNING_KEY"]

# Then change your urlpatterns to override the Anymail Mailgun webhook views:
urlpatterns = [
    # ...
    # Change this:
    #   url(r'^anymail/', include('anymail.urls')),
    # to this:
    url(r'^anymail/mailgun/inbound(_mime)?/$', 
        MailgunInboundWebhookView.as_view(api_key=MAILGUN_WEBHOOK_SIGNING_KEY), 
        name='mailgun_inbound_webhook'),
    url(r'^anymail/mailgun/tracking/$', 
        MailgunTrackingWebhookView.as_view(api_key=MAILGUN_WEBHOOK_SIGNING_KEY), 
        name='mailgun_tracking_webhook'),
    # ...
]

[dominik-lekse]

Many thanks @mbk-ok and @medmunds for investigating and providing a suitable workaround that quickly

With regard to the scenario, I confirm that I ran into this issue after a key rotation in the Mailgun API security. After the key rotation, I have replaced the ANYMAIL_MAILGUN_API_KEY with the updated key and discovered that the inbound mail webhook failed with 400 Bad request due to a seemingly wrong signature. Further, I did not notice that in the initial account configuration both Mailgun API keys are the same and that they differ after the first key rotation. As a developer, I would not expect this.

We will go short-term with the proposed workaround until an updated release of anymail with the new setting. In particular, there is no need for getting both keys to an equal state again.

Further feedback from a developer point of view:

Although I am very familiar with the class-based views in Django and despite I have looked into the source of MailgunBaseWebhookView during my investigation, I have not noticed that the configuration from Django settings in the Anymail view classed can be overridden. A reason for this is certainly that this logic of the configuration precedence is hidden in api_key = get_anymail_setting('api_key', esp_name=self.esp_name, kwargs=kwargs, allow_bare=True). Actually this is a great feature and allows using multiple Mailgun accounts and API key on different webhook endpoints in the same Django project. This topic along with some example like in the workaround you provided could be a additional section in https://anymail.readthedocs.io/en/stable/tips/#tips-tricks-and-advanced-usage.

[medmunds]

MAILGUN_WEBHOOK_SIGNING_KEY setting released in Anymail v6.1.