How to read server statistic correctly

[yockee]

I need a better understanding about reading server stats.

I have like a total of 10-12 map services. All hosted on AWS, single server.

I monitor all of them by reading their Total Request, Maximum Running Instances, Average Response Time and Maximum Response Time.

Here is the data:

My biggest question is how on earth "Map Svc A"with Total Request of 831 at Time 1 has Average response time of under 1 second whereas "Map Svc B" with Total Request of just 31 has max Response Tme of 221 seconds ? At time 4, all  both Map Services A and B are really slow. 

What is going on ? How to read the server stats correctly ?

Here is some graphics to give general idea.

The graphic below shows the Max Running Instances, Total Request and Average Response of ALL Map services at both times, 1 and 4.

The graphic below shows that Service Maximum Running Instances all 0.

Here is the graphic about Total Request of Map Service A and B at two different times:

Here is a graphic about Average Response Time:

Any explanation welcome. I want to understand it better..

Thanks

[HenryLindemann]

Hi @yockee 
Total Request: This is the volume of hits you have e.g. how may views of the map service

Maximum Running Instances: Is the longest time one of these requests took to complete, this can be a long running request if the service enters the que.

Average Response Time: this is just the Average response over all the requests 

The reason for the big discrepancy in time will be because SVC B is probably a complex service compared to SVC A or SVC A can have a cache, for example a boundary of a state will be less complex than a boundary of a neighborhood what we call a cadaster. or SVC A can be a MAP Service and SVC B can be a feature service which is an editable service and takes a lot more time to draw. Or SVC B's data source can be a database in SQL Server and SVC A might be a file on disk called a File Geodatabase.

What you can have a look at is if the service is dynamic or dedicated dynamic pooling is where all the services share the instances normally set at 4 and they start slower but use less RAM, whereas dedicated are faster but consume more RAM. you can also increase the dynamic pooling to a max of your CPU count, and this will increase concurrency.

Around the Cache this you normally build if the service is a static service

Hope that helps.

Kind Regards
Henry

[berniejconnors]

Another thing to consider is that some requests just take longer because they are requesting more data.  For example, a map service that displays civic address points - a request that returns 10 points will be faster than a request that returns 10,000 points.  As an ArcGIS Administrator, it is hard to imagine what the users will attempt to request from your services.  You should be able to get log files of your inbound traffic to examine the requests.  Pay special attention to the requests that take more time.  These logs should ba available from the web server on your web adaptor (not the ArcGIS Server logs).  These would likely be IIS logs or Apache logs.

If you have published ArcGIS web apps (like an Experience Builder or Web App Builder app) you can also use the Chrome Dev tools to look at the requests generated by the web app.  I made the mistake of publishing a Web App Builder app with a poorly configured Search widget.  I configured the search widget to use "LIKE" instead of "=".  Using "LIKE" negates the benefits of the indexes you have built for your attribute data and causes your searches to be slower.  You should also ensure you are only searching agasinst one or two attributes - not all attributes.  Looking at your inbound traffic will really help identify problems.

There are tools in ArcGIS Pro to help you optimize the performance of your map before you publish it as a map service.

If you only have 12 map services, your server should be able to handle publishing all 12 map services as dedicated services.  Your most popular map services should be configured with higher min and max instances.  This will help avoid requests waiting to be processed.  The pooled map service can be saved for your less popular map services.  The stats your are collecting will show you which map services get fewer requests.

Bernie.

[berniejconnors]

Esri's low-code / no-code app builders make it really easy to build apps that look very polished.  But looking polished does not equate to fast and efficient use of your ArcGIS Server services.  You still have to ensure the apps are configured properly or you will have poor performance >> slow apps and busy servers.

[yockee]

@berniejconnors , @HenryLindemann

Okey. Thanks for the reply.

Being hit by heavy Total Request, why the Service Maximum Running Instance lie flat 0 for both Map Services A and B ? Shouldn't they be picking up the request ? Or may be there is no available instances to process the request ? I set the instances to a fixed number. For Map Service A : min 2 max  5; Map Service B : min 2 max 4.

By the way, Service Maximum Running Instance is the number of instances that you set to handle the request. Those map service A and B are feature services and being accessed by mobile apps and desktop, possibly concurrently, but not all the time. 

[berniejconnors]

I rarely look at the Maximum Running Instance stats.  I have never found it useful and it does not appear to align with the other stats I am seeing.  We use very few feature services - we use map services and image services.  Here are charts of our most popular map service.  This service depicts our tax parcels (~450,000 polygons) and it receives about 270,000 hits per day.  It is configured for min=1 and max=6 instances and it is running on a site with three machines.  So the true min is 3 and the true max is 18.

This was captured at 9 am on a Thursday morning showing the last 24 hours. You can see the running instances drops to 1 during the quietest part of the day and is frequently at 6 during the busiest part of the day.  I don't understand why your Max Running Instances is showing 0 - it should be showing 2 if your feature services are using dedicated instances with min = 2.

And don't forget the number of running instances are tied to several timeout settings.  Once an instance is activated it will hang around in an idle state until the timeout value expires:

You asked a similar question last summer.  Perhaps you should contact Esri support for clarification.

Bernie.