A partial hardware failure at our data center disabled multiple internet connections. Because it was a partial failure it did not trigger the automatic fail-over process from that provider. Our application monitors caught the issue immediately and technical teams started working on parallel tracks to switch off the internet connection from our data center provider and to help restore the internet connection. The data center provider restored full connectivity prior to us routing traffic through a redundant connection. They are working with the hardware vendor to prevent this from happening in the future and we are reducing latency within the process fail-over to the redundant connection.