Running "Check for Content Updates" made all services unresponsive

[MelissaBrooks]

Context

Our Open Data items reference services from an External ArcGIS Enterprise Portal. 

After changing the service url of 19 Open Data items I ran the "Check for Content Updates" in the Content Library.

(as per https://community.esri.com/community/gis/web-gis/arcgis-hub/blog/2019/12/09/how-to-manually-refresh-... )

Issue

The effect of this was massive, 

- All Portal Services (server and imagery) became unresponsive for approximately 2 hours

- This impacted all public facing map products (Main GIS Viewer, District Plan, B4UDIG service etc)

- Also impacted internal products that referenced our public basemaps

- the server manager and administrator sites were unresponsive for this time

- I could RDP into the server fine

- double-ups in our Open Data Catalog for the items that had updates to their service url (expected, but it has now been 3 days and it has VERY slowly brought the count back down to normal - 2 outstanding as-at this post)

Summary

I'm tending towards the webadapter being inundated with requests from HUB (given that basemaps on our imagery server were also affected), but am asking for any feedback as to the reason using this tool would render our servers dead in the water so to speak for a good couple of hours. 

Additional

We had some SyncToolsEx issues but will wait for a few days to see if this comes up again now the Catalog is settling back down.

Take-home

I expected a few minutes interruption to services but next time will definitely do this out-of-hours (if at all, as I'm now hesitant to use this function given the unexpected impact it has had)

I'm happy to provide more information to explain more about what we observed on Friday (NZ time) if necessary

Cheers

Melissa

Melissa Brooks

Senior GIS Analyst

New Plymouth District Council

New Zealand

https://geohub.npdc.govt.nz

Clinton Ballandis‌ 

Patrick Hammons‌

Graham Hudgins‌

[ThomasHervey1]

I'm sorry to hear this. What you experienced is certainly unexpected behavior. We will investigate the cause of the issue shortly. The "Check for Content Updates" tool is intended to check for updates your content and reindex public content that hasn't yet been updated automatically. The process kicks off a job that possibly put too much stress on your server. Would you mind emailing me at thervey@esri.com and send a link to a few of the 19 items that were updated? 

[MelissaBrooks]

Cheers Thomas, I've just sent something through.