topic Re: GeocodeServer - ArcSOC using a lot of memory on each search in ArcGIS Enterprise Questions

GeocodeServer - ArcSOC using a lot of memory on each search

MaximeDemers — Thu, 30 Jan 2020 15:01:25 GMT

I have discovered that our GeocodeServer ArcSOC are using up to 17Go of memory per search. This is quite a lot in my opinion. Our server machine has 44Go of RAM in total and from time to times we can see some insufficient memory errors in the logs caused by a GeocodeServer instance.

Here is a screen shot taken from the task manager when a search request was made to a GeocodeServer.

I see in the .loc file there is a property named RuntimeMemoryLimit. It is setted to 512 MB but clearly this not limiting the memory usage.

We are using ArcGIS Server 10.6 on windows 2012 r2 server.

Is it normal? Is there something we can do to prevent this?

Re: GeocodeServer - ArcSOC using a lot of memory on each search

JoshuaBixby — Thu, 30 Jan 2020 16:56:16 GMT

Given the processor is at 12%, I am guessing you have an 8-core machine and one of the cores is being maxed out by the geocoding request. If a single geocoding request maxes out a core for more than a second or two, I would say something isn't quite right with the geocoding. It is hard to say whether what you are seeing is "normal" without knowing more about the geocoder and geocoding service.

Re: GeocodeServer - ArcSOC using a lot of memory on each search

AndrewWhitman1 — Thu, 30 Jan 2020 17:17:22 GMT

Greetings Maxime,

I think there are a few things that may be helpful for managing resources while running ArcGIS Server at version 10.6. The version you are running does not yet support pooled instances which was introduced at 10.7. If you ever do upgrade to 10.7, this is the very first recommendation I would make.

However, I would like to know how many cores are on the machine you are using for the geocoding server. It would also be helpful to know what number of maximum and minimum instances are being assigned to the geocoding service within Server.

You can set these values based on the amount of resources needed. For instance, if I had 4 cores on this geocoding machine, I would set the maximum to 5. I would follow the n +1 formula where n=number of cores.

This geocoding service has 1 instance running, but I can change this by clicking on the service and editing the max and min instances.

I also found a few resources that can help you to manage resources on this machine and reduce unnecessary memory usage.

GIS Server Wiki

http://www.wiki.gis.com/wiki/index.php/Server_Software_Performance#Server_process

Inside the ArcGIS Server Site

https://enterprise.arcgis.com/en/server/latest/administer/windows/inside-an-arcgis-server-site.htm#ESRI_SECTION1_693C429E8C06449085778507602D3403

Geoprocessing Service Settings

https://enterprise.arcgis.com/en/server/10.6/publish-services/windows/geoprocessing-service-settings-pooling-and-processes.htm

I hope this helps!

Re: GeocodeServer - ArcSOC using a lot of memory on each search

MaximeDemers — Thu, 30 Jan 2020 17:50:19 GMT

Hi Joshua,

yes it is an 8-core machine that we have. I am not sure to understand the relation between the cpu usage and the memory consumption by an ArcSOC process. I can see the total cpu usage going from 2% to 18-20% on each geocoding request.

I also notice that the request is often really slow, like 10-12 seconds and sometimes really fast less than 1 second.

Does providing the .loc file would help?

Re: GeocodeServer - ArcSOC using a lot of memory on each search

MaximeDemers — Thu, 30 Jan 2020 17:55:45 GMT

This is a 8-core machine and there is 1 minimum instance and 2 maximum instance for the service.

Does providing the .loc file could helps? I have noticed also that sometimes a request is really slow (10-12 seconds) and sometimes really fast (0.25-1 second)

Re: GeocodeServer - ArcSOC using a lot of memory on each search

AndrewWhitman1 — Thu, 30 Jan 2020 18:04:10 GMT

Hi Maxime,

Can we try increasing the maximum number of instances to 9 for the geocoding service? Lets keep the .loc file where it is at for now and see if increasing resources helps with this issue.

After increasing, can we test geocoding to see what the performance looks like?

I also found this ArcSoc optimizer tool, which may be useful in the use case. I have not had the chance to use it but it could be worth a try.

ArcSoc Optimizer Tool

https://www.arcgis.com/home/item.html?id=a0c2dde661f14b8ca58b8b7b5ddeb379

I hope this helps.

Re: GeocodeServer - ArcSOC using a lot of memory on each search

MaximeDemers — Thu, 30 Jan 2020 18:10:50 GMT

Hi Andrew,

Thanks for your help here.

Unfortunately, increasing the maximum instances number changes nothing. I just tested it a few minutes ago. That's what I expected, because doing a single request use only one instance and only one ArcSOC is created by instance.

The locator is a Single Line Input. If I do a search using "123" for input, it takes 10-12 seconds and then I see the memory of the ArcSOC going up to 17GB.

If I do a search using "1234" for input, the request resolve in 44ms and there is no peek in the memory usage.

Re: GeocodeServer - ArcSOC using a lot of memory on each search

AndrewWhitman1 — Thu, 30 Jan 2020 18:15:42 GMT

Okay, it seems like there is more to this than just resource usage. It could be helpful to open up a support ticket with Esri Support Services. It is hard to say definitively since each request is responding differently.

I can say that 17GB of memory is way more than I would expect for such a simple operation.

Re: GeocodeServer - ArcSOC using a lot of memory on each search

AndrewWhitman1 — Thu, 30 Jan 2020 18:24:49 GMT

HI Maxime,

Does the locator you are using have the "suggest" option enabled? I am wondering if there is a defect related to the way the locator is setup.

Can we test with the suggest function disabled within the locator?

Re: GeocodeServer - ArcSOC using a lot of memory on each search

RandallWilliams — Thu, 30 Jan 2020 18:50:24 GMT

What version of ArcGIS Server are you working with? I know of a few memory leaks related to geocoding that have since been addressed in 10.7.1.

Re: GeocodeServer - ArcSOC using a lot of memory on each search

MaximeDemers — Thu, 30 Jan 2020 18:55:18 GMT

Andrew,

This is specifically the suggest method that cause the problem. The ArcGIS Server version is 10.6.1.

Something weird is that I have published the same locator on another ArcGIS Server v10.6.1 that we use for development and there is no problem at all. The ArcSOC does not use more than 120MB of memory and the request resolve in 0 to 0.5 second maximum.

Re: GeocodeServer - ArcSOC using a lot of memory on each search

MaximeDemers — Thu, 30 Jan 2020 18:57:33 GMT

Hi Randall,

It's the version 10.6.1

Re: GeocodeServer - ArcSOC using a lot of memory on each search

RandallWilliams — Thu, 30 Jan 2020 19:06:14 GMT

Do have this patch installed?

ArcGIS Server 10.6.1 Geocode Suggestions Leak Patch

Re: GeocodeServer - ArcSOC using a lot of memory on each search

AndrewWhitman1 — Thu, 30 Jan 2020 19:12:00 GMT

Randall and I are traveling down the same line of thought. If this is the defect we are both thinking of, there is a patch. Check out his post below.

Re: GeocodeServer - ArcSOC using a lot of memory on each search

MaximeDemers — Thu, 30 Jan 2020 19:17:32 GMT

Randall,

Thank you for pointing it out! We will definitely install the patch to see if it solve the problem!

Re: GeocodeServer - ArcSOC using a lot of memory on each search

MaximeDemers — Thu, 06 Feb 2020 18:41:03 GMT

I confirm that the patch solved the leak. Thank you!