GeocodeServer - ArcSOC using a lot of memory on each search

[MaximeDemers]

I have discovered that our GeocodeServer ArcSOC are using up to 17Go of memory per search. This is quite a lot in my opinion. Our server machine has 44Go of RAM in total and from time to times we can see some insufficient memory errors in the logs caused by a GeocodeServer instance.

Here is a screen shot taken from the task manager when a search request was made to a GeocodeServer.

I see in the .loc file there is a property named RuntimeMemoryLimit. It is setted to 512 MB but clearly this not limiting the memory usage.

We are using ArcGIS Server 10.6 on windows 2012 r2 server.

Is it normal? Is there something we can do to prevent this?

[JoshuaBixby]

Given the processor is at 12%, I am guessing you have an 8-core machine and one of the cores is being maxed out by the geocoding request.  If a single geocoding request maxes out a core for more than a second or two, I would say something isn't quite right with the geocoding.  It is hard to say whether what you are seeing is "normal" without knowing more about the geocoder and geocoding service. 

[MaximeDemers]

Hi Joshua,

yes it is an 8-core machine that we have. I am not sure to understand the relation between the cpu usage and the memory consumption by an ArcSOC process. I can see the total cpu usage going from 2% to 18-20% on each geocoding request.

I also notice that the request is often really slow, like 10-12 seconds and sometimes really fast less than 1 second.

Does providing the .loc file would help?

[AndrewWhitman1]

Greetings Maxime, 

I think there are a few things that may be helpful for managing resources while running ArcGIS Server at version 10.6. The version you are running does not yet support pooled instances which was introduced at 10.7. If you ever do upgrade to 10.7, this is the very first recommendation I would make. 

However, I would like to know how many cores are on the machine you are using for the geocoding server. It would also be helpful to know what number of maximum and minimum instances are being assigned to the geocoding service within Server. 

You can set these values based on the amount of resources needed. For instance, if I had 4 cores on this geocoding machine, I would set the maximum to 5. I would follow the n +1 formula where n=number of cores. 

This geocoding service has 1 instance running, but I can change this by clicking on the service and editing the max and min instances. 

I also found a few resources that can help you to manage resources on this machine and reduce unnecessary memory usage. 

GIS Server Wiki

http://www.wiki.gis.com/wiki/index.php/Server_Software_Performance#Server_process

Inside the ArcGIS Server Site 

https://enterprise.arcgis.com/en/server/latest/administer/windows/inside-an-arcgis-server-site.htm#E...

Geoprocessing Service Settings

https://enterprise.arcgis.com/en/server/10.6/publish-services/windows/geoprocessing-service-settings...

I hope this helps!

[MaximeDemers]

This is a 8-core machine and there is 1 minimum instance and 2 maximum instance for the service.

Does providing the .loc file could helps? I have noticed also that sometimes a request is really slow (10-12 seconds) and sometimes really fast (0.25-1 second)

[AndrewWhitman1]

Hi Maxime, 

Can we try increasing the maximum number of instances to 9 for the geocoding service? Lets keep the .loc file where it is at for now and see if increasing resources helps with this issue. 

After increasing, can we test geocoding to see what the performance looks like?  

I also found this ArcSoc optimizer tool, which may be useful in the use case. I have not had the chance to use it but it could be worth a try. 

ArcSoc Optimizer Tool

https://www.arcgis.com/home/item.html?id=a0c2dde661f14b8ca58b8b7b5ddeb379

I hope this helps.  

[MaximeDemers]

Hi Andrew,

Thanks for your help here.

Unfortunately, increasing the maximum instances number changes nothing. I just tested it a few minutes ago. That's what I expected, because doing a single request use only one instance and only one ArcSOC is created by instance.

The locator is a Single Line Input. If I do a search using "123" for input, it takes 10-12 seconds and then I see the memory of the ArcSOC going up to 17GB.

If I do a search using "1234" for input, the request resolve in 44ms and there is no peek in the memory usage.

[AndrewWhitman1]

Okay, it seems like there is more to this than just resource usage. It could be helpful to open up a support ticket with Esri Support Services. It is hard to say definitively since each request is responding differently. 

I can say that 17GB of memory is way more than I would expect for such a simple operation.  

[AndrewWhitman1]

HI Maxime, 

Does the locator you are using have the "suggest" option enabled? I am wondering if there is a defect related to the way the locator is setup. 

Can we test with the suggest function disabled within the locator? 

[MaximeDemers]

Andrew,

This is specifically the suggest method that cause the problem. The ArcGIS Server version is 10.6.1.

Something weird is that I have published the same locator on another ArcGIS Server v10.6.1 that we use for development and there is no problem at all. The ArcSOC does not use more than 120MB of memory and the request resolve in 0 to 0.5 second maximum.