Looking for performance troubleshooting suggestions

MichaelMorisette

Hello

We are running ArcGIS Enterprise 10.9.1 on Windows Server 2016 Standard (single machine deployment) with all available patches installed.

We have been experiencing intermittent performance issues for the past several months.  These performance drops appear to have no correlation with usage, and seem to randomly happen at any given time of the day.

We have been working with our IT network team and, thus far, all of our available tools tell us everything is fine in terms of the server/network load.  We were able to install ArcGIS Monitor (2023.2), which has given us some metrics quantifying how bad the performance gets but we're still unable to identify the cause.  I have attached a graph from Monitor showing the average request response time versus request rate for all services over the past 7 days.

Does anyone have suggestions where to go with this in terms of pinpointing the issue?

berniejconnors

Michael,

        Are your users noticing performance problems?  If all you have is some graphs showing some slow response times you might still be satifying 99% of your users.  And just 1% of your users are doing stupid things and causing their own problems / slow responses.

        To look into the problem I would look at the web adaptor logs.  My ArcGIS Server deployment has the web adaptors running on a separate Linux server under Apache.  On your single machine deployment your web adaptor is likely running under IIS but I am not certain.  Go into the IIS logs and at look at the incoming requests.  The web adaptor log file should include the duration of the request.  Examine the long running requests and identify what it is your user was trying to do.  The web adaptor log records should include the REST URL of the request.  Are they requesting a lot of data?  Are they trying to match a search term against all attributes in a map service instead of just querying against a single attribute?  Is there one service that is causing most of the slow requests?  Are all of your slow requests coming from the same web app?  Is someone trying to scrape data from your map services?  You should be able to get a lot of info from your web adaptor logs.

        I find the web adaptor logs very useful for identifying long running requests.  Every day I run a report to identify all requests that exceed 60 seconds.  I usually have to examine about 100 records each day.  We get close to 1 million requests each day.  Most of the time I just look at the time stamp on the request:

• At midnight our services recycle. 
• At 1 am our server reboots. 
• At 2 am we update several map services with fresh data. 

        If I see a slow request outside of these times I will look more closely and try to determine why the request was slow.  I'll copy the ArcGIS REST URL from the log file and run it in my browser.  I'll examine all of the parameters in the URL and determine what the user was trying to do.  This helped me identify a Web App Builder app with a poorly configured Search widget.  It was one that I had configured myself.

I hope this helps,

Bernie.