Email Reporting and Metrics Data Delay
Incident
Report for Anthology - Encompass
Postmortem
Our engineers have identified that the Root Cause of the issue that prevented Email Reporting Data from updating was due to bad data related to the size of the data being passed in from a single customer environment. Code changes were added to account for this situation so that it should not impact the updating of reporting again.
Resolved
Our engineers have confirmed that the system has caught up with processing all email data for reporting purposes. It has been determined that this issue was not caused by the code changes in the release from last night; however during the troubleshooting, we did role back changes that were made for the release. These changes do need to be reapplied to the production environment. We are planning to reapply these release items during a maintenance effort that will be posted shortly. Thank you for patience as we worked through this issue and your continued partnership.
Monitoring
Our engineers have identified the root cause for the issue and have implemented a fix for the issue. We are seeing reporting data now coming into the system. This will take some time to process however we are seeing it moving. We will continue to monitor the system very closely as it works through all of items that need to be processed. We will follow up with more information in 2 hours or sooner if more information is available.
Update
Our Engineers are continuing to investigate the lack of data being reflected in email reporting metrics in Encompass. We do continue to see Email Messages sending and delivering as intended at this time. If you try to send an A/B campaign you will see issues with the delivery of the Winner message as there as there is a dependence on reporting to be able to determine the winner message. As further information becomes available updates will be provided here. We Thank You for your patience as we continue to investigate.
Update
Our Engineers are continuing to investigate the delays in email reporting metrics coming into the Encompass. We continue to see Email Messages sending delivering as intended at this time. As further information becomes available updates will be provided here. We Thank You for your patience as we continue to investigate.
Investigating
We are experiencing a delay in email reporting and metrics data and are working to remediate the issue. An update to status will be made as soon as email reporting is caught up.
This incident affected: Production Sites (US Production Sites, Canadian Production Sites) and Email Marketing (Email Reporting).