Arconline Issues?

[TiffanyRufo]

IS anyone else having trouble with ArcOnline and/or Survey123 loading maps and layers? It seems to take a lot longer and the 'trouble adding layer to the map. Still trying' pop-up is showing up quite frequently?

Just wondering if this is resultant from the update/change (It looks fresh btw) or if I need to purse this internally.

Thanks!

[ThomasColson]

Seems to be working now. I have less confidence in ArcGIS Online Health Dashboard .  as of yesterday morning it was showing "Healthy" when things clearly weren't working. I'm guessing there's no after-hours monitoring of this service, as the first ESRI confirmation of the outage coincided at around 7 AM west coast time? 

[KellyGerrow]

Hi Thomas,

We do have after hours monitoring of our services, unfortunately, this issue occurred specifically during PST/EST work hours. If you encounter an issue with ArcGIS Online outside of regular business hours (6:00 am - 5:00 pm PST), submit a case through MyEsri, that indicates the Issue Category "ArcGIS Online Outage". This will be directed to our 24/7 team, Esri Accounts .

Here is some additional information about the problem that was experienced and actions that we are taking to improve monitoring for the future.

Between 7am PST on June 27th and Noon PST on June 29th a subset of our customers experienced short periods of sporadic timeouts and errors with their hosted feature services during PST/EST work hours. 

The issue was due to an application triggering excessive resource consumption within the hosted feature services implementation causing slow performance and time outs to a subset of hosted feature services. 

Because ArcGIS Online’s hosted feature services are distributed over multiple data centers,  the impact was limited to a subset of customers.

The troubleshooting of the issue, which was unrelated to the deployment of our June release during the same timeframe, took time because of the volume of traffic and telemetry, the intermittent nature of the application and its overlapping in time with the June release. Throughout the duration of this issue we had a team monitoring the affected clusters and trying to stabilize the systems as quickly as possible while we were investigating the root cause. 

Even with this in place, there was an impact to some customers.

Once the cause was isolated, the problem was analyzed and an update to the hosted feature services implementation was rolled out to handle the condition and stop it from reoccurring. Going forward we are working on improving our test coverage for this and similar issues while also enhancing our toolsets and monitoring to make sure we can detect and fix any future issues quickly. 

Thanks to everyone who helped notify us of the issue and for your patience while it was investigated.

-Kelly

[JosephAlberts]

Was working fine this morning, up until about 5 minutes ago. Now I'm see slow/no responses on hosted feature layers again.

[JosephAlberts]

Update: It looks like they're operating normally again for now.

[JosephKerski]

I was having Survey123 issues earlier this week so thanks for confirming.  Will try it again today.

[TiffanyRufo]

So far so good at the City of Sandusky

[TiffanyRufo]

I know that an update was recently done on the 20th, I am experiencing very slow Feature layer drawing as well as a lovely little orange exclamation point symbol on all of my maps, that says:

The layer, Manhole_Collections_COS, was not drawn completely because it has too many features to draw. Try zooming in to a smaller area.

This shows up on my layers with 6,000 features as well as my layers with 20 features. Just wondering if this is related to the update or isolated. Anyone else experiencing this?

[AdrianWelsh]

Tiffany,

It might be an issue similar to last time. Looking at this status site:

ArcGIS Online Health Dashboard 

There were issues yesterday. They seem to be resolved but since there was a new roll out of ArcGIS Online, maybe some issues are still lingering (or new ones are popping up)? I know the ArcGIS Online team is working on it though!

[TiffanyRufo]

Thanks for the reply Adrian,

There aren't many GIS users in my organization that use it quite as much as I do so it's always hard to tell if it's just me or not. Just wanted to make sure.

Thanks again!

[MikeMinami]

Tiffany,

Can you share the URL to your layer (with 6000 features)? Is this a layer hosted on AGOL or pointing to your own ArcGIS Server?

The exclamation point is not new, we just made it more prominent looking. It indicates that AGOL was not able to retrieve all features in the layer typically because there are too many features and our request for features exceeds the limit on the server to return features. However, 6000 features doesn't seem like a lot, so there could be some timeouts that are causing the queries to fail.

Mike