On Tuesday, December 4, 2019, some emails sent between 2:10pm and 2:45pm (CST) were delayed. During this period emails that were previously scheduled to send and emails that were released to send immediately were delayed. These emails appeared stuck in the queued tab until services were restored to normal operations. After services were restored, most emails were processed and delivered successfully, however a handful of email campaigns were transitioned to the failed to send status (with a red exclamation point on the history tab).
At 2:10pm (CST), technical teams were alerted to an email that was delayed while getting member token information. After initial investigation to determine the reason for the delay, it was determined that the recovery logic within the email system had put the email into a loop. The decision was made to fail this email by 2:40pm and the email delivery system was sending emails normally by 2:50pm. After this email was failed the email delivery system returned to normal operations. Further investigation identified a configuration change to prevent this from happening in the future.