SYSTEM STATUS

Encompass Blocker Severity 1 - ENC-39237- Some emails and previews from the 2pm hour are stuck in the send process
Incident Report for Anthology - Encompass
Postmortem

On Tuesday, December 4, 2019, some emails sent between 2:10pm and 2:45pm (CST) were delayed. During this period emails that were previously scheduled to send and emails that were released to send immediately were delayed. These emails appeared stuck in the queued tab until services were restored to normal operations. After services were restored, most emails were processed and delivered successfully, however a handful of email campaigns were transitioned to the failed to send status (with a red exclamation point on the history tab).

At 2:10pm (CST), technical teams were alerted to an email that was delayed while getting member token information. After initial investigation to determine the reason for the delay, it was determined that the recovery logic within the email system had put the email into a loop. The decision was made to fail this email by 2:40pm and the email delivery system was sending emails normally by 2:50pm. After this email was failed the email delivery system returned to normal operations. Further investigation identified a configuration change to prevent this from happening in the future.

Posted Dec 04, 2019 - 19:11 CST

Resolved
All emails that were stuck in the sending process have either been sent or failed. In email history you will see a green check for those emails sent successfully and a red exclamation point for those that failed. Any emails that partially sent (red exclamation point with an "actual" sent count) can be resent using the sent list from email reporting as an exclusion list.
Posted Dec 03, 2019 - 15:41 CST
Investigating
Some emails and previews from the 2pm hour are stuck in the send process has been identified as a Severity 1 Blocker Defect and will be reviewed and fixed by our development team as soon as possible. The ticket number for your reference is ENC-39237. If the email has a red exclamation point in history you can attempt to resend it. For emails that are still spinning the technical teams are working to resolve this issue, locate and fail any problematic emails in order to reduce any delays in sending.
Posted Dec 03, 2019 - 14:39 CST
This incident affected: Email Marketing (Email Home) and In Application Issues - Severity 1 Defects.