Duplicate Events created
Incident Report for Chargify
This incident has been resolved.
Posted about 3 years ago. May 06, 2016 - 08:11 CDT
From May 04 2016 20:10 (EDT) until May 05 2016 22:10 (EDT) Chargify incorrectly may have generated multiple Event records for a single payment. The event records duplicated are all of the type `payment_success` or `payment_failure`. The events are all identical (same payment, same metadata) except for the event ID. Since events also create webhooks, you may have received multiple webhooks all tied back to a single payment.

After an extensive search, we have found no cases where a customer charged twice. *ONLY* the events and webhooks are duplicated.

We're very sorry for any trouble or confusion this causes you. Although we have detailed tests to ensure that we never accidentally charge someone twice, those tests weren't able to catch the sequence that led to these duplicated events. We have fixed the race condition that caused this bug and have carefully tested other possible code paths that might have resulted in similar bugs. We're now confident that such a problem like this won't happen again.

After considering the options, we have decided *not* to delete the duplicate webhooks or events. Many merchants have already seen the events and they may be used as part of automated systems that are relying on those IDs to exist and never be removed. However, if you would prefer that we remove the duplicates to clean up your event history, we'll be glad to help. Just email support@chargify.com and we'll take care of it.

Again, we're very sorry. Data integrity and reliability are the most important things we do at Chargify and we take it very seriously. So we'll continue to review our processes and try even harder to provide the best service we can. Thanks for being a Chargify customer!
Posted about 3 years ago. May 06, 2016 - 08:11 CDT