Hi Thomas,
We do have after hours monitoring of our services, unfortunately, this issue occurred specifically during PST/EST work hours. If you encounter an issue with ArcGIS Online outside of regular business hours (6:00 am - 5:00 pm PST), submit a case through MyEsri, that indicates the Issue Category "ArcGIS Online Outage". This will be directed to our 24/7 team, Esri Accounts .
Here is some additional information about the problem that was experienced and actions that we are taking to improve monitoring for the future.
Between 7am PST on June 27th and Noon PST on June 29th a subset of our customers experienced short periods of sporadic timeouts and errors with their hosted feature services during PST/EST work hours.
The issue was due to an application triggering excessive resource consumption within the hosted feature services implementation causing slow performance and time outs to a subset of hosted feature services.
Because ArcGIS Online’s hosted feature services are distributed over multiple data centers, the impact was limited to a subset of customers.
The troubleshooting of the issue, which was unrelated to the deployment of our June release during the same timeframe, took time because of the volume of traffic and telemetry, the intermittent nature of the application and its overlapping in time with the June release. Throughout the duration of this issue we had a team monitoring the affected clusters and trying to stabilize the systems as quickly as possible while we were investigating the root cause.
Even with this in place, there was an impact to some customers.
Once the cause was isolated, the problem was analyzed and an update to the hosted feature services implementation was rolled out to handle the condition and stop it from reoccurring. Going forward we are working on improving our test coverage for this and similar issues while also enhancing our toolsets and monitoring to make sure we can detect and fix any future issues quickly.
Thanks to everyone who helped notify us of the issue and for your patience while it was investigated.
-Kelly