On Tuesday, December 4, 2019, the email delivery service was delayed between 8:30am and 9:50am (CST). During this period emails that were previously scheduled to send and emails that were newly released to send immediately were delayed. Prior to 9am most emails continued to send normally, however after 9:00am an increased number of emails were delayed. These emails appeared stuck in the queued tab until services were restored. After services were restored most emails were processed and delivered successfully, however a small number of email campaigns were transitioned to the failed to send status (with a red exclamation point on the history tab). Most of these failed emails were updated immediately, however some were updated by 12:40pm.
At approximately 8:30am (CST) technical teams received an alert that the email delivery system became backed up in the process that translates audience information into Encompass member information. Technical teams initially resolved a performance bottleneck with the process that translates Encompass Segments into Member IDs. After additional investigation, a bottleneck was discovered in the process that generates token and link information for individual members. The decision was made to fail a small group of emails that had been delayed for more than 30 minutes, which returned the system to normal operation. Further investigation determined that the process was blocked by a few emails with large segments and a large number of tokens. Monitoring tools were immediately put in place to identify emails that had the potential to delay the delivery process. Additional code and configuration changes are planned to prevent this from happening.