SYSTEM STATUS

Encompass Blocker Severity 1 - ENC-39222- Some emails and previews from the 8AM hour and after are stuck in the send process
Incident Report for iModules Software
Postmortem

On Tuesday, December 4, 2019, the email delivery service was delayed between 8:30am and 9:50am (CST). During this period emails that were previously scheduled to send and emails that were newly released to send immediately were delayed. Prior to 9am most emails continued to send normally, however after 9:00am an increased number of emails were delayed. These emails appeared stuck in the queued tab until services were restored. After services were restored most emails were processed and delivered successfully, however a small number of email campaigns were transitioned to the failed to send status (with a red exclamation point on the history tab). Most of these failed emails were updated immediately, however some were updated by 12:40pm.

At approximately 8:30am (CST) technical teams received an alert that the email delivery system became backed up in the process that translates audience information into Encompass member information. Technical teams initially resolved a performance bottleneck with the process that translates Encompass Segments into Member IDs. After additional investigation, a bottleneck was discovered in the process that generates token and link information for individual members. The decision was made to fail a small group of emails that had been delayed for more than 30 minutes, which returned the system to normal operation. Further investigation determined that the process was blocked by a few emails with large segments and a large number of tokens. Monitoring tools were immediately put in place to identify emails that had the potential to delay the delivery process. Additional code and configuration changes are planned to prevent this from happening.

Posted Dec 04, 2019 - 19:10 CST

Resolved
All emails that were stuck in the sending process have either been sent or failed. In email history you will see a green check for those emails sent successfully and a red exclamation point for those that failed. You can resend any emails that failed with a red exclamation point if you still want those emails to send.
Posted Dec 03, 2019 - 12:44 CST
Monitoring
The queue is clearing and emails are starting to send normally. Some emails from the 8 and 9am hours are still waiting to be sent. If the email has a red exclamation point in history you can attempt to resend it.
Posted Dec 03, 2019 - 10:24 CST
Investigating
Some emails and previews from the 8AM hour and after are stuck in the send process - has been identified as a Severity 1 Blocker Defect and will be reviewed and fixed by our development team as soon as possible. The ticket number for your reference is ENC-39222. If the email has a red exclamation point in history you can attempt to resend it. For emails that are still spinning the technical teams are working to resolve this issue, locate and fail any problematic emails in order to reduce any delays in sending.
Posted Dec 03, 2019 - 09:41 CST
This incident affected: Email Marketing (Email Home) and In Application Issues - Severity 1 Defects.