Email Marketing Outage
Incident
Report for Anthology - Encompass
Postmortem
On Wednesday morning, a third party provider of the email editor experienced an issue with their AWS cluster. Our internal team, shut off the service so no emails would send without content. The third party provider quickly implemented a workaround internally and the services were brought back online. Our internal teams then initially reconnected the services in our QA environment and did testing to confirm that the issue was fully resolved. Once that was confirmed, we reconnected the production environment. Our third party provider has now permanently implemented the workaround so that in the event that this occurs again, they can immediately switch over.
Resolved
This incident has been resolved.
Monitoring
At 9:03am CST, our third party provider provided a resolution to this issue. Our internal teams then restarted our services to reconnect and conducted testing. The sends that were delayed have sent successfully, as well as the 10am CST sends. We will continue to monitor this and post an update shortly.
Identified
Our 3rd party provider has identified the issue and are working on a fix. We will provide additional updates as they become available.
Investigating
Our engineers have identified an issue with a 3rd party service that is affecting email marketing, including email sending. We have reached out to our 3rd party service and are awaiting an update. We will provide additional updates as they become available.
This incident affected: Email Marketing (Email Home, EDM, Email Marketing Deliverability (non Sendgrid)).