Encompass Severity 1 Blocker - ENC-29858- Secure and admin pages are intermittently erroring
Incident
Report for Anthology - Encompass
Postmortem
It has been determined that Thursday's (9/21) intermittent website errors were the result of a software failure of a traffic routing appliance (load balancer). The load balancing failure resulted in an abnormal amount of traffic being routed incorrectly to a small subset of servers, causing website errors and page load failures. The redundant load balancer was activated and is running as expected. Future maintenance will be scheduled to perform any necessary adjustments. At this time, we are working to find a permanent solution to the issue with the load balancer and will communicate a new date for the reinstatement of the 9/20 release which reactivated caching improvements originally attempted in July.
Resolved
This issue has been resolved.
Update
We are aware that many clients were still experiencing issues with their Encompass sites receiving errors or simply not loading even after we rolled back last night's release. At this point, we have taken further steps to resolve the issue and are currently seeing improvements. We will continue to monitor the issue and provide additional updates.
Monitoring
While we have not identified the true cause of the intermittent site errors/page load failures, we have completed the roll back of last night's release and are continuing to investigate the true cause. Sites should be returning to normal and we will continue to provide updates.
Investigating
Secure and admin pages are intermittently erroring and not loading - has been identified as a Severity 1 Blocker Defect and will be reviewed and fixed by our development team as soon as possible. The ticket number for your reference is ENC-29858.
This incident affected: Production Sites (US Production Sites, Canadian Production Sites) and In Application Issues - Severity 1 Defects.