A summary of the issue from yesterday:
Thank you for your patience as we worked through performance and publishing issues with certain ArcGIS Online hosted features layers yesterday. We understand the urgency and importance of continuing to provide a resilient, redundant and well architected system and we are confident that everything is back to normal and no data was lost.
What happened:
Between 3:00AM and 6:45 PM PST Tuesday September 4, 2018 ArcGIS Online experienced continual intermittent problems with a subset of our infrastructure that provides hosted feature services and spatial analysis.
Why did it happen:
The issue was due to a weather/lightning event which caused electrical problems and a shutdown of part of the infrastructure with the Microsoft Azure South Central US region. This infrastructure shut down resulted in a subset of ArcGIS Online Services being unavailable for publishing or viewing.
Our Response:
The ArcGIS Online operations team worked with Microsoft Azure during the event to diagnose the problem and then began work on swift recovery. During the investigation and recovery, the status of the issue was updated at status.arcgis.com until restoration was verified and tested. In parallel to the above activity we also triggered our Disaster Recovery process for restoring affected services in a different region from database backups. The recovery of the original affected data center completed earlier.
Additional notes and plans:
- Our services already run in High Availability mode covered by the SLA of Azure and take advantage of redundancy and fail over across multiple instances. We will investigate and consider additional improvements once the root cause analysis from Microsoft has been released to strengthen the availability of our current deployments.
- We will also look at improvements to the communication process to insure you have the most reliable and updated information possible.
