All feature layers suddenly unable to load

[ChristinaHoddinott1]

I came into work this morning ready to collect some more data using the Collector app. However, when I opened the appropriate map, none of the layers were able to load. I found this to be strange, so I went back to my desktop to see what was going on. In AGOL now NONE of my hosted feature layers are able to load data. See the attached picture to see the message I am getting. What in the world is going on? Everything was working perfectly last week!

I do typically download the data from AGOL on a regular basis, but I did not for one particular feature layer after collecting data on Friday (what can I say, I was ready for the weekend). So unfortunately I will have to go out a recollect all the data I collected last Friday, which is not cool.

What could possibly be going on? Am I going to have to start from scratch with all my feature layers and their corresponding maps?!

Initially, I published all maps from Arc Desktop as this was the easiest way to do so and preserve my desired symbology. Then if I need to update a layer, I append the data to the appropriate layer using the Update Feature tool in AGOL. I do sometimes update a feature layer through Desktop publishing, because for whatever reason, if I change the shape of a feature in Desktop, it does not update it using only the Update Data tool on AGOL. 

This is extremely frustrating, as you can probably imagine. I need to be able to count on the products being used to WORK. Hopefully someone can shed some light as to what is going on and that all is not lost...

[ChristinaHoddinott1]

Okay, got a look at the AGOL Health Dashboard and there seem to be system-wide issues here. (I had no idea this existed! Three pages deep into Google and I stumbled upon it.)

[MichaelVolz]

Anyone from ESRI know when Hosted Feature Services from AGOL will be available again?

[KellyGerrow]

Hi Michael,

Our operations team is currently investigating and working to resolve the issue. Please keep an eye on http://status.arcgis.com for updates and I will also update this post. On the Organization page of your ArcGIS Online organization, you will also be able to see the status of the affected systems.

Thanks and we are working to get this resolved as soon as possible.

-Kelly 

[KellyGerrow · Accepted solution]

A summary and update of the issues yesterday

Thank you for your patience as we worked through performance and publishing issues with certain ArcGIS Online hosted features layers yesterday.   We understand the urgency and importance of continuing to provide a resilient, redundant and well architected system and we are confident that everything is back to normal and no data was lost.

What happened:

Between 3:00AM and 6:45 PM PST Tuesday September 4, 2018 ArcGIS Online experienced continual intermittent problems with a subset of our infrastructure that provides hosted feature services and spatial analysis. 

Why did it happen:

The issue was due to a weather/lightning event which caused electrical problems and a shutdown of part of the infrastructure with the Microsoft Azure South Central US region. This infrastructure shut down resulted in a subset of ArcGIS Online Services being unavailable for publishing or viewing.

Our Response:

The ArcGIS Online operations team worked with Microsoft Azure during the event to diagnose the problem and then began work on swift recovery.  During the investigation and recovery, the status of the issue was updated at status.arcgis.com until restoration was verified and tested.  In parallel to the above activity we also triggered our Disaster Recovery process for restoring affected services in a different region from database backups.  The recovery of the original affected data center completed earlier.

Additional notes and plans:

• Our services already run in High Availability mode covered by the SLA of Azure and take advantage of redundancy and fail over across multiple instances. We will investigate and consider additional improvements once the root cause analysis from Microsoft has been released to strengthen the availability of our current deployments. 
• We will also look at improvements to the communication process to insure you have the most reliable and updated information possible.