Apps deployed in ArcGIS Enterprise 11.4 not loading when users accessed simultaneously

[jantolihao]

Hi, everyone!

Our team is currently experiencing issue where our apps deployed in ArcGIS Enterprise are not loading properly when accessed by multiple users simultaneously. Some layers in our apps are taking too long to load and eventually throws a data source error and an error 500 when we checked the developer tools. (Please see screenshots below)

We also checked the Portal logs and encountered "The hosting server may not be reachable." message. (Please see screenshot below)

We have not experienced this issue before where multiple users accessed simultaneously. We just encountered this issue since yesterday.

Here are some of the workarounds that we did in attempting to resolve the issue, but still no luck:

• Monitored the server CPU and RAM, but all seem stable

• Restarted services consumed by apps in ArcGIS Server Manager

• Restarted all ArcGIS Enterprise components in Windows Services

• Already installed necessary patches

• We also performed speed test and connection seems fine

By the way, our apps are deployed in ArcGIS Enterprise 11.4 which is in AWS cloud.

Has anyone encountered this issue before?

I appreciate any ideas on how to resolve this. Thank you very much.

[Brian_M]

When you mention monitoring the CPU and RAM, can you clarify what percentage of utilization we're looking at? For example, were they consistently under 50%, or were they approaching 90% on the ArcGIS Server machine?

It seems to me that this might be an issue where ArcGIS Server is struggling to handle the high volume of requests from multiple users accessing the applications simultaneously. Do you know if ArcGIS Server crashes or becomes unresponsive during these times?

To investigate further, I recommend reviewing the ArcGIS Server logs and searching for keywords like "timeout" or "error performing query". This can help identify if any services are timing out. From there, you can determine if these affected services are tied to the applications being used. If they are, you might need to adjust the service settings, such as increasing the number of instances, or switching services from shared to dedicated. But remember to consider the available RAM and memory on the ArcGIS Server machine when making these changes.

I will link to some documentation that goes more in-depth on this. 

[jantolihao]

Hi, @Brian_M.

Thank you for responding. Regarding the CPU and RAM, percentage is below 50% for both (around 20-30%). We already checked the ArcGIS Server settings and switched some services to dedicated instances. We also adjusted the maximum number of records returned by Server settings of the services that are lagging. We have not found any solution for this yet. We have not encountered this error before where we simultaneously accessed the apps. We just experience this recently.

[David_McRitchie]

Random idea, and might not be specific to our use case here, but do we have any Databases or folders registered in the Server Manager? If so then could we try a manual restart of the DynamicMappingHost service within the Server Manager?

Theres a recent bug that's come up for 11.4 that is preventing a number of arcsoc processes from starting after the machine has been restarted. Your memory usage being at 30% makes me think this could be the case.

Hope that helps,

David

[jantolihao]

Hi, @David_McRitchie.

Thank you for the response. We already restarted the DynamicMappingHost service within ArcGIS Server Manager. However, the problem still persists. We are still active in looking for possible solution for this.  

[David_McRitchie]

Thanks for checking. Can I just double check if this is all working reliably if we just have one user accessing the service?

It might be worth publishing a new dataset and adding this to a map, seeing if we get any errors when we open one instance of Map Viewer on this, and then try this with an additional user.

If we get issues with a second user accessing the service then could we check the Basemap configuration? I have seen this issue before when using customised WMTS services as basemaps. The resolution in these cases was to ensure that anything created in the Map Viewer was only used on JavaScript v4.x applications (such as Experience Builder, Dashboards etc, not Web App Builder or Classic Map Viewer).

Maybe that will fix this?

David