It has been determined that Thursday's (9/21) intermittent website errors were the result of a software failure of a traffic routing appliance (load balancer). The load balancing failure resulted in an abnormal amount of traffic being routed incorrectly to a small subset of servers, causing website errors and page load failures. The redundant load balancer was activated and is running as expected. Future maintenance will be scheduled to perform any necessary adjustments. At this time, we are working to find a permanent solution to the issue with the load balancer and will communicate a new date for the reinstatement of the 9/20 release which reactivated caching improvements originally attempted in July.
Posted Sep 26, 2017 - 10:31 CDT
This issue has been resolved.
Posted Sep 22, 2017 - 09:41 CDT
We are aware that many clients were still experiencing issues with their Encompass sites receiving errors or simply not loading even after we rolled back last night's release. At this point, we have taken further steps to resolve the issue and are currently seeing improvements. We will continue to monitor the issue and provide additional updates.
Posted Sep 21, 2017 - 11:52 CDT
While we have not identified the true cause of the intermittent site errors/page load failures, we have completed the roll back of last night's release and are continuing to investigate the true cause. Sites should be returning to normal and we will continue to provide updates.
Posted Sep 21, 2017 - 10:45 CDT
Secure and admin pages are intermittently erroring and not loading - has been identified as a Severity 1 Blocker Defect and will be reviewed and fixed by our development team as soon as possible. The ticket number for your reference is ENC-29858.
Posted Sep 21, 2017 - 09:28 CDT
This incident affected: Production Sites (US Production Sites, Canadian Production Sites) and In Application Issues - Severity 1 Defects.