SendGrid smtp-id not always matching anymail_status

[joshkersey]

ESP: SendGrid
django-anymail: 2.2
Python: 2.7
Django: 1.11.10

Just after we make the msg = AnymailMessage(**kwargs) call we store the data from msg.anymail_status in our database for tracking. The issue we're seeing is that the SendGrid webhook will often times return POST data with a message_id and esp_event['smtp-id] that do not match the values Anymail generated for us.

As an example:
We send a message and receive an smtp-id of <20180523175141.98405.51520@example.com> in the msg.anymail_status.
When the webhook communicates updates to us we find that the message_id for the same message is <YTJPMXaQSY-KzuEUmXrDWg@ismtpd0025p1las1.sendgrid.net> which means that we are unable to identify which message this belongs to.

Here's the code that calls AnymailMessage:

text_message = render_to_string(templates['text'], context)
html_message = render_to_string(templates['html'], context)
msg = AnymailMessage(
    subject=templates['subject'],
    body=text_message,
    to=[user.email],
)
msg.attach_alternative(html_message, "text/html")
msg.send()

# store the anymail status data to our database for tracking via the webhook
Email().create_from_anymail(msg.anymail_status, Email.INVITATION)

The Email() model holds the create_from_anymail(anymail_status, event) method where event is an optional custom tracker for the event triggering the email send (unrelated to anymail):

@classmethod
def create_from_anymail(cls, anymail_status, event=None):
    for recipient in anymail_status.recipients:
        kwargs = {
            'to_email': recipient,
            'status': anymail_status.recipients[recipient].status,
            'message_id': anymail_status.recipients[recipient].message_id,
            'response': anymail_status.esp_response
        }

        if event:
            kwargs['event'] = event

        email = Email(**kwargs)
        email.save()

From what I can gather from the SendGrid webhook documentation the smtp-id should be returned for every webhook POST and is a unique ID attached to the message by the originating system.

I've also contacted SendGrid support but wanted to present the issue here in case anyone has seen something similar and may have ideas on how to get a consistent identifier from our system into SendGrid.

[medmunds]

Anymail by default* generates a new message_id for each message it sends through SendGrid. It sets that in the message's Message-ID header, which is what SendGrid is supposed to pass back to webhook calls as the smtp-id field.

Two questions:

1. Can you check an actual sent message as received in your inbox, and see what Message-ID header it ended up with? An Anymail-generated Message-ID should look like <numbers@your-from-email-domain>. If it does, but you're getting a different smtp-id in the webhook, that sounds like a SendGrid bug. If it doesn't, then SendGrid has overridden the Message-ID header from Anymail, and we'd need to figure out why. (Also, there should be exactly one Message-ID/Message-Id header, case-insensitive.)

2. Your example code doesn't provide a from_email for the msg, so Django's DEFAULT_FROM_EMAIL setting will be used. Have you set that to something that matches a domain you've authenticated with SendGrid? (Related: when you say you "receive an smtp-id of <20180523175141.98405.51520@example.com> in the msg.anymail_status", the "example.com" part was really your actual sending domain, and you just redacted it here, right?)

I'll try to check later to see if SendGrid changed something out from under us, but my best guess right now is they're maybe overriding Message-ID with their own value under some undocumented conditions.

* I'm assuming you've already seen Anymail's docs on SendGrid Message-IDs, and that your settings don't override Anymail's default SENDGRID_GENERATE_MESSAGE_ID=True.

[joshkersey]

1. I've checked all of our test emails from the past couple of days and they do show the Anymail ID in the value for the Message-ID. Also, Message-ID only appears one time in each email.
2. Correct, we are using a DEFAULT_FROM_EMAIL in our settings and, yes, the "example.com" in the smtp-id is a redacted domain. The redacted domain has been validated via SendGrid.

I will relay any useful information I receive from SendGrid support.

[joshkersey]

One additional note, the SendGrid engagement events (click and open) have been consistently sending the correct (Anymail) smtp-id.

[medmunds]

OK, I'm able to reproduce. For me, at least, this definitely seems related to whether the From email uses a sender domain that's authenticated with SendGrid (https://app.sendgrid.com/settings/sender_auth). If the From domain is not authenticated, the "smtp-id" sent to the webhooks is different from the Message-ID header in the sent/received message.

Details...

When I send messages from an authenticated, DKIM-validated domain in my SG account...

From: from@test-sg.anymail.info
Message-ID: <152712480034.85680.5099186793774837923@test-sg.anymail.info>

... the webhook payload has the correct smtp-id:

  "smtp-id": "<152712480034.85680.5099186793774837923@test-sg.anymail.info>"

BUT when I send messages from @example.com, which is (obviously) not an authenticated domain for my SG account, I see exactly the behavior you described: the received message's Message-ID header matches what Anymail sent...

From: from@example.com
Message-ID: <152712433591.85282.8340115595767222398@example.com>

... but the webhook payloads have a totally different smtp-id:

  "smtp-id": "<-g6-N0riQ3WNhvp9aBJpfQ@ismtpd0006p1sjc2.sendgrid.net>"

Please let me know what you hear back from SendGrid support. (I could sort of understand if they were trying to prevent un-authenticated sender domains from appearing anywhere in sent mail headers, but since they're leaving @example.com in the Message-ID and the From header, that's not what's going on.)

If anyone from SendGrid engineering stops by here while investigating, the examples above are not redacted; feel free to pull them up in your logs. Also, that incorrect "smtp-id" from the from@example.com webhook payload actually does appear in the message headers, as an "HTTP id":

Received: from MjM2NTg4Nw ([REDACTED:sending-machine-ip-address])
 by ismtpd0006p1sjc2.sendgrid.net (SG) with HTTP id -g6-N0riQ3WNhvp9aBJpfQ
 Thu, 24 May 2018 01:12:16.237 +0000 (UTC)

Again, though, it doesn't make any sense that this HTTP id would be substituted for the actual Message-ID value in the webhook 'smtp-id'.

[medmunds]

One additional note, the SendGrid engagement events (click and open) have been consistently sending the correct (Anymail) smtp-id.

This is because SendGrid doesn't actually provide smtp-id with the payloads for those events, but Anymail has a trick to work around this using SendGrid's custom_args.

[joshkersey]

I've checked and double checked and even confirmed with SendGrid support that our sender validation is setup properly for our sending domain.

I discussed the webhook issue at length with SendGrid support yesterday explaining that we see the proper Message-ID in the email headers but that both Message-ID and smtp-id are being overwritten in the Webhook POST data for some, but not all, webhook responses.

Here's the response I received from SendGrid support:

Both "smtp-id" and "Message-ID" are automatically posted by SendGrid when relaying the message, which might be causing your unique message id to not set properly. Can you try setting your unique message ID within the header as something other than 'Message-ID' - like SAmsgid (or something like that) to see if it relays properly?

Following their request I modified theensure_message_id() function in anymail/backends/sendgrid.py by adding an additional key in custom_args:

def ensure_message_id(self):
    """Ensure message has a known Message-ID for later event tracking"""
    if "Message-ID" not in self.data["headers"]:
        # Only make our own if caller hasn't already provided one
        self.data["headers"]["Message-ID"] = self.make_message_id()
    self.message_id = self.data["headers"]["Message-ID"]

    # Workaround for missing message ID (smtp-id) in SendGrid engagement events
    # (click and open tracking): because unique_args get merged into the raw event
    # record, we can supply the 'smtp-id' field for any events missing it.
    self.data.setdefault("custom_args", {})["smtp-id"] = self.message_id
    self.data["custom_args"]["anymail-id"] = self.message_id

After this update I receive the anymail-id in the esp_event consistently for each message and was able to also update anymail/webhooks/sendgrid.py to pass the anymail-id in as message_id for the AnymailTrackingEvent() so the signals I'd written would continue to work.

I'm happy to put together a PR with these changes if you'd like but wanted to follow up with the information I received and a report on what has worked to resolve the issue for my project.

[medmunds]

Thanks for tracking this down and for the PR. Using our own custom_args field is a much better approach than trying to work around SendGrid's unpredictable smtp-id and Message-ID behavior. (Also, past experience indicates we could wait a very long time for SendGrid to fix an acknowledged API bug.)

Since the anymail_id is only used for matching up tracking events, I'm sort of inclined to just switch it to something like a uuid, and get rid of all the backend code that mucks with the Message-ID header. We don't generate Message-ID for any other backends. (And with many ESPs, the webhook tracking id is not the Message-ID—in fact, it's often a uuid generated by the ESP.)

Any concerns with that? I'm trying to figure out if there's any way this might be a breaking change. (Anymail message_id is supposed to be treated as an opaque string, because there are so many different ESP formats.)

[joshkersey]

Definitely support the UUID approach and the simplification it brings. I think it could be a nearly invisible rework as long as we flow the UUID through message_id for both the AnymailStatus() and SendGridTrackingWebhookView() so current implementations can carry on. I don't currently have a project utilizing SendGrid API v2 so that may need some testing.

[medmunds]

OK, I'm sold. I can either merge your PR and then make the UUID switch separately, or if you're up for it you could update the PR.

My memory is the SendGrid v2 backend has exactly the same code (I think it originated pre-v3), and could have the same changes. (This also reminds me it's probably time to deprecate that backend.)

[joshkersey]

Happy to make the UUID updates while I'm in there making the updates requested for the PR.

[medmunds]

Fix released in Anymail v3.0. Thanks again for the contribution.