topic Re: In which sense the pooling settings affect the performance of displaying services when consumed? in ArcGIS Enterprise Questions

In which sense the pooling settings affect the performance of displaying services when consumed?

JamalNUMAN — Sun, 30 Dec 2018 21:19:07 GMT

In which sense the pooling settings affect the performance of displaying services when consumed?

For example, for the service indicated in the screenshot below, “the maximum number of instances per machine” is set to be 7. Does this mean that the if we have 9 CONCURRENT (simultaneous) users, then 7 of them will be able to display the service while the other two will be waiting?

Thank you

Best regards

Jamal

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

ModyBuchbinder — Mon, 31 Dec 2018 05:55:59 GMT

A display request should take less the 1 second.

If 9 requests will come in the same second then 7 will be processed by 7 SOC's and the other 2 will wait for one second.

In real life if the display is tune up then 2 instances will be more then you need.

If a user ask for 4 pan/zoom operation in a minute then 10 users create 40 requests in a minute and 1 SOC is about 66% busy. In that rate even one should do the work.

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

abdullahqasrawi — Mon, 31 Dec 2018 09:07:50 GMT

Hi Mody

As I understand from you that the maximum response time will not exceed one second for displaying service, but I monitor my services by statistics tool and I found that the maximum response time for some services is more than that, please see screenshot below that show the response time is about 30 second for my service.

What do you think?

Best

Abdullah

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

ModyBuchbinder — Mon, 31 Dec 2018 09:15:33 GMT

Very hard to say by this information.

Not all requests must be display, if you did some query or run GP tool it might take much longer.

If the machine was busy doing something else (for example copy a lot of data to the disk that the service is using) it can slow things.

From your graph it looks like the system is fine most of the time. You need to find out what happened in these pick times.

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

abdullahqasrawi — Mon, 31 Dec 2018 09:53:33 GMT

Thanks for your reply

Actually, I am monitoring my Server also, and I feel that everything going well, screenshot below show normal values when High response time happened for the services.

What do you expect?

Best
Abdullah

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

JamalNUMAN — Mon, 31 Dec 2018 21:45:33 GMT

Many thanks Mody for the input.

It is really very confusing why displaying services in our web mapping application gets slow at sometime during the day. The Windows resources monitor doesn’t indicate any up normal usage for CPU, RAM, HD or Network.

The other issue is that, we couldn’t figure out how the number of requests are associated to:

Response time

ArcSoc.exe

CPU, RAM, HD, Network

In other words, we have no definite clue why ArcGIS Server fails to display services with an acceptable response time.

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

ModyBuchbinder — Tue, 01 Jan 2019 05:44:04 GMT

You had 26500 requests in half an hour. That is almost 900 a minute. This is why your response time is high.

There is no way for even 100 users to create such load (9 request per user per minute?). You need to find what created this load.

Possibly a program somewhere that create this load.

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

JamalNUMAN — Tue, 01 Jan 2019 07:33:12 GMT

Many thanks for the prompt answer Mody

Our services are open to public through the link below:

http://geomolg.ps/services.html

Now, we decided to lock all of services and thus users are exclusively able to display them through our web mapping application.

http://geomolg.ps/

Also, these services are downloadable through our application as they are all configured in the ArcGIS Open data
Further, users are used to consume cached imageries and extract them using QGIS!

I’ll be monitoring and see if this will decrease the request.

But eventually, how the number of Requests are related to the number of ArcSOC available? Does increasing the min\max number of ArcSoc help?

We observed also that the cached imagery has low number of requests despite the fact that it is the most displayed one! How this can be explained?

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

JohnCuthbertson1 — Tue, 01 Jan 2019 07:35:41 GMT

One possible explanation for long response times (20+ seconds) is that your mapservices are starting/stopping. This process can take multiple seoconds (and even minutes if you have lots of layers). You can check to see if this is happening by reviewing the ArcGIS logs (a subject in itself) or you could check 'Task Manager' on the ArcGIS server to see when each ArcSOC started. The configuration that works for us is to set MIN/MAX Instance to the same value and to set the services to recycle every 24 hours between midnight and 1AM. If we get any Mapservice recycling during the day we investigate root cause and correct.

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

JamalNUMAN — Tue, 01 Jan 2019 09:23:46 GMT

Many thanks John for the input.

- Most of our services are set as 1 for min and 2 as max

- The recycle is set to be 24

- I don’t see anything unusual in the logs

- May you please help me how to go to the Task Manager of ArcGIS Server?

What other things I need to consider?

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

JohnCuthbertson1 — Wed, 02 Jan 2019 04:16:28 GMT

Sorry 'Task Manager' (WIndows Task Manager) does not display the time that each process started (My mistake).

You can get this information via Powershell

Get-WmiObject win32_process | Select-Object Name, ProcessId, WorkingSetSize, HandleCount, ThreadCount, CreationDate, KernelModeTime, UserModeTime, Commandline |Out-GridView

In your case all the ArcSOC CreationDates should be around midnight last night. If there are other start times then, while that start was occuring, there will be an elongated response for the request that caused the start and potentially elongated responses for other requests that arrive while the start is being actioned.

To chase why you have start times away from midnight you need to look at the ArcGIS logs

Firstly to verify that you are capturing the logs you should be able to find the following entries around midnight

Datetime	Type	Code	Message
28/12/2018 0:00	INFO	8000	Instance of service 'Utilities/PrintingTools' has been created successfully.
28/12/2018 0:00	INFO	8000	Instance of service 'System/PublishingToolsEx' has been created successfully.
28/12/2018 0:00	INFO	8000	Instance of service 'System/SpatialAnalysisTools' has been created successfully.
28/12/2018 0:00	INFO	8000	Instance of service 'System/SpatialAnalysisTools' has been created successfully.

If you cannot see them then you probably have logging set to 'Warning' or 'Severe'. Try setting the level to 'Info' . There is a performance impact doing this, but we run with it set to 'FINE' and we process over 2 million requests per day without significant impact.

We run with 'FINE' so that we can get request response times..

Datetime	Source	MethodName	Type	Code	Elapsed
28/12/2018 0:00	Utilities/Geometry.GeometryServer	/simplify	FINE	100004	0.0029
28/12/2018 0:00	Utilities/Geometry.GeometryServer	/simplify	FINE	100004	0.0008
28/12/2018 0:00	External/Imagery_External.MapServer	WMSServer.GetToPost	FINE	100002	0.00064
28/12/2018 0:00	External/Imagery_External.MapServer	WMSServer.get_Data	FINE	100004	0.57157
28/12/2018 0:01	Utilities/Geometry.GeometryServer	/simplify	FINE	100004	0.00162
28/12/2018 0:01	Utilities/Geometry.GeometryServer	/simplify	FINE	100004	0.00093
28/12/2018 0:01	External/Mineral_External.MapServer	WMSServer.GetToPost	FINE	100002	0.00056
28/12/2018 0:01	External/Mineral_External.MapServer	WMSServer.get_Data	FINE	100004	0.15819
28/12/2018 0:01	External/Imagery_External.MapServer	WMSServer.GetToPost	FINE	100002	0.00035
28/12/2018 0:01	External/Imagery_External.MapServer	WMSServer.get_Data	FINE	100004	0.47298
28/12/2018 0:01	External/Imagery_External.MapServer	WMSServer.GetToPost	FINE	100002	0.00066
28/12/2018 0:01	External/Imagery_External.MapServer	WMSServer.get_Data	FINE	100004	0.21022
28/12/2018 0:01	External/Mineral_External.MapServer	WMSServer.GetToPost	FINE	100002	0.00109

You can get similar response data from the IIS logs on the server. As an aside, in regards to your query on tiles, we see the request in the IIS logs, but do not see any stats in the ArcGIS logs. My (uneducated) assumption is that ESRI decided to bypass any logging for TILES in the interests of making it as fast as possible.

We have actually gone to the effort of collecting the IIS and ArcGIS logs into Splunk and then displaying the overheads visually..

The left column is from the IIS logs, the right column is from the ArcGIS logs

The top row is showing the number of requests, the bottom row is showing how long those requests took.

The tile counts in the top left hand chart are the most significant, but become insignificant from an elapsed perspective (and are not shown at all from an ArcGIS perspective).

We did all this work on earlier versions of ArcGIS, hopefully the instrumentation provided by ESRI in the newer versions is more useful (although nothing beats looking at the raw data yourself)

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

ModyBuchbinder — Wed, 02 Jan 2019 06:10:29 GMT

ArcGIS Monitor ArcGIS Monitor | Optimize Your Enterprise GIS is the best tool for you.

It cost money and it is not simple to install and set but it will give you full information about what each service is doing, how much CPU, memory etc each service takes.

You should look into it.

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

JoshuaBixby — Wed, 02 Jan 2019 16:01:54 GMT

There could be factors outside ArcGIS Server that affect response times. For instance, when my organization migrated to a new data center, we would have periods of time where server response times were slower. For all the troubleshooting we did, we could not figure out why the slow periods were occurring for months. We finally determined that a network interconnect in the data center between our servers and the rest of the world, both internal and external networks, was incorrectly sized when the data center was built out. The slow periods were do to what other groups were doing on the network, and the switch would get so bogged down that server response times went up across the board.

Beyond exploring your network topology and performance more, I would also take a deeper dive into specific requests that are running during the slow periods. If you have some users, scripts, etc... that are running slow-running REST calls; those could tie up SOCs and cause slower response times for simple map requests. We had an issue where a web app was configured to filter results to a specific fiscal year, but yet the field with the fiscal year was not indexed. With millions of records, a full-table scan was being run each time people would hit the service using that web app. When the app got used by a lot of users, we had so many SOCs constantly hitting that same feature class and doing full-table scans, the response times would creep up. I was surprised the response times stayed under 5 seconds, but I guess that it a testament to how fast computers are these days.

What I constantly push back on in my organization is that ArcGIS Server cannot turn poorly tuned data or configured map documents into lightning fast GIS services. It is usually under load when the issues start to show themselves.

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

JamalNUMAN — Fri, 04 Jan 2019 13:54:27 GMT

Thank you guys for the useful input.

I have already commented with the following on another thread.

This issue here is that there is no math equation based on which one can decide the min\max number of instances (ArcSOC.exe) needed for each published service.

In a test environment, we have been trying to understand service performance when executing actions such as panning, zooming, quiring, etc. We found out that, for example, zooming a service that contains a layer of 300,000 polygons is really a heavy action and would consume around 12% of the CPU (machine specs: 3GHz CPU, 4 Cores, 64 GB RAM, 2133 MHz). This means that if 10 users are zooming at the same time, then the CPU will get full just from zooming action on a service!

Zooming actions consumes on a service 12% of the CPU

Each ArcSoc.exe occupies around 80 MB on the RAM

Then imagine that we have 100 or 1000 users are zooming at the same time. Does this lead to the fact that no any server what the specs are will be sufficient to take care of services?

The issue gets complicated to be explained if we wanted to know how actions (such as pan, zoom, query, etc.) are translated in terms of requests, and then to figure out the number of ArcSoc.exe needed to serve these requests\actions

This could explains why our web mapping application fails at peak hours!

http://geomolg.ps/

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

JoshuaBixby — Fri, 04 Jan 2019 18:02:00 GMT

Does this lead to the fact that no any server what the specs are will be sufficient to take care of services?

If by "any" you mean one, a single server, then the answer is yes (or do I mean to say no). To rephrase, it is common to need more than one GIS server, which is why Esri supports a multi-machine deployment model.

In my organization, our ArcGIS Server site that hosts the highest-demand GIS services is made up of 6 GIS servers and 2 web adaptors. Typically all 6 machines run around 40% total CPU utilization spiking to 90% at times.

Have you run the MXD through PerfQAnalyzer: New 10.6/10.6.x Version (Build 173) Available for Download ? Slow redraw times from the data tier will translate through to slow redraw times in the GIS service. There are numerous performance-related configuration aspects to GIS services themselves. How is this service configured, e.g., are you caching at all ?

Re: In which sense the pooling settings affect the performance of displaying services when consumed?

JamalNUMAN — Fri, 25 Jan 2019 20:33:31 GMT

Thank you Joshua for the help. The points you have indicated are very useful

Best

Jamal