Emails, webhooks, and subscription processing currently delayed
Incident Report for Chargify
Postmortem

On Friday March 3rd, beginning at 16:28 EST we discovered an unusually high level of spam emails being sent with the Chargify platform.

We first temporarily disabled email sending and were able to then target and stop the offending emails. (This is the action that caused the delay in background processing on Friday from 16:30 EST to 18:20 EST).

In order to ensure maximum deliverability, we also switched to delivering all new emails from our published backup IP address (167.89.80.59 instead of 167.89.19.147). This way legitimate emails from Chargify merchants would not be marked as spam.

Even so, every recipient server has different rules on what emails it considers spam and some may still be weary. Hotmail, AOL, and Yahoo are the most aggressive at flagging messages and we've had a few reports of legitimate messages from Friday not making it through.

However at this time we believe the impact is minimal and we're seeing very good deliverability rates on all the metrics available to us, especially on new emails being sent out today. We'll continue to monitor the situation very closely to keep ensuring that our sending reputation remains good and as many emails as possible get delivered.

If you're seeing some of your emails marked as spam, we recommend the following steps:

  1. Mark the email as "not spam" in your email client.
  2. Check to make sure you've configured your domain's SPF to list Chargify as a permitted sender: https://help.chargify.com/communications/mail-deliverability.html

If you have any questions please feel free to reach out to support@chargify.com. We'll be happy to help investigate!

Posted 9 months ago. Mar 05, 2018 - 12:29 CST

Resolved
All webhooks, emails, and subscription processing are up-to-date.

The only outstanding issue we are tracking is that _certain test sites_ are not allowed to send email at this time. This is a preventative measure to protect trusted sites. We will follow up on this with separate updates, since most merchants and sites are now not impacted by any issues.
Posted 9 months ago. Mar 02, 2018 - 18:19 CST
Identified
We are about to re-enable sending of webhooks, live-site emails, and all subscription processing.

Sending of test site email for _some sites_ will still be restricted until further notice.
Posted 9 months ago. Mar 02, 2018 - 17:30 CST
Investigating
We have had to disable a portion of our background job processing, which affects the sending of emails, sending of webhooks, and subscription processing.

You will experience a delay in each of these functions until we resume sending.

Updates will be posted periodically.
Posted 9 months ago. Mar 02, 2018 - 17:04 CST