topic Re: 10.5.1 Memory Usage in ArcGIS Enterprise Questions

10.5.1 Memory Usage

DaveTenney — Tue, 21 May 2019 20:15:53 GMT

All,

i was curious if anyone else encountered an interesting behavior once upgrading desktop and server to 10.5.1....

We have noticed that after upgrading and running the "manage map server cache tiles", esri ultimately maxes out the memory, and this usually happens 90min after starting the tool. We have run this process many times on the data and not until after we upgraded to 10.5.1 are we now experiencing this.

Machine Specs

Intel Xeon CPU E5-4620 v2

Windows Server 2012 R2

16gb Ram

4 Cores

thanks,

dave

Re: 10.5.1 Memory Usage

George_Thompson — Tue, 21 May 2019 20:23:21 GMT

What version where you on before the upgrade?

How many services are on the machine?

Re: 10.5.1 Memory Usage

DaveTenney — Tue, 21 May 2019 20:36:04 GMT

10 services total: 2 cached map service, 1 dynamic map, 7 locator services

we were running 10.2.2, when we upgraded we rebuilt all the services manually using 10.5.1. we were able to rebuild the services manually just fine. now we create an area of interest that is built from a delta check and we get to about level 16 out of 19 and the cachingtools begin consume massive amounts of memory.

we start seeing entries in the cachingtools logs that mention "out of memory", then the process fails. with a similar message:

Data access failure, layer = Image, error string = Could not access data for layer or table Image during drawing§TilesWorker: Data access failure, layer = Image, error string = Could not access data for layer or table Image during drawing§TilesWorker: Data access failure, layer = Image, error string = Could not access data for layer or table Image during drawing Failed to cache extent:........

Re: 10.5.1 Memory Usage

George_Thompson — Tue, 21 May 2019 20:41:44 GMT

Thanks for that info, how many instances on the caching services?

How many instance are running on the other 10 services?

I have seen other posts (cannot find right now) that users had something similar with newer versions consuming more memory than previous ones.

Is the machine (VM or physical) actually out of memory when the error occurs?

Re: 10.5.1 Memory Usage

MichaelSchoelen — Tue, 21 May 2019 20:43:32 GMT

Could it be a setting with the pooling parameters?

Re: 10.5.1 Memory Usage

DaveTenney — Tue, 21 May 2019 20:54:21 GMT

Caching services were set to a max of 3, we decided to take the n-1 approach (n = # of cores) in order to prevent the resources from being maxed out on the machine. As for all the services, they run at a max of 2.

yes, i can sit and watch the gptool process run at the same time tracking the machine stats and before too long, the memory just starts jumping up like crazy and ultimately maxing out.

Re: 10.5.1 Memory Usage

DaveTenney — Tue, 21 May 2019 21:11:37 GMT

Michael,

i doubt it is pooling as we have taken a conservative approach with the caching services.

we are running the gptools at a max of 3 on a 4 core machine, we did this because we did not want to take the chance to max out the machine. unfortunately, we are now seeing something that runs wild with the available memory.

thanks

dave

Re: 10.5.1 Memory Usage

George_Thompson — Wed, 22 May 2019 12:08:08 GMT

Thanks for that information and that is a great approach. My general thoughts based on that information is that the machine may need more RAM after the upgrade. Would it be possible to increase the RAM to 32GB and test the workflow again?

I know that you mention the OS as WS2012R2, it is also updated to at least the April 2017 patch?

Re: 10.5.1 Memory Usage

MichaelVolz — Wed, 22 May 2019 12:57:39 GMT

What kind of data is changing in your problematic cached mapservice (e.g. parcels, road centerlines, other)?

How often are you upating the cache?

On average, how many features are changing per cache update?

What is your largest scale that you are caching? Can you see if the problem still occurs if you don't include the 3 or 4 largest scales, just as a test?

Is this process occurring when there is low or no load on the AGS server?

Dynamic mapservices perform much better in 10.5.1 so you might consider removing layers from cache and make them dynamic.

Re: 10.5.1 Memory Usage

DaveTenney — Wed, 22 May 2019 13:23:36 GMT

here is the part in which i get a little confused by everything...

we can run the process manually on level 18 and 19 and recreate all tiles just fine. the minute we utilize an area of interest for those same levels is when we see the memory usage skyrocket.

we had to prove this process in 2 other environments before we could even implement all this in production. this is a behavior we did not see in any of those 2 environments.

im not sure where the patches stand for the OS.

Re: 10.5.1 Memory Usage

DaveTenney — Wed, 22 May 2019 13:30:43 GMT

1) we have roads, addresses, other individual point, and line features.

2) this update process runs on a weekly basis

3) on average we see 20k-50k changes total

a. this is why we do the delta b/c so little changes that we don't want to waste time rebuilding everything

4) when we rebuild tiles we do it in two steps

a. forced scales - these are level 0-15, very little data total so we just rebuild them all (usually only 10min)

b. selective scales - this is levels 16-19, this is where we implement our area of interest based on the delta results

1) it is within the selective scales that we see the memory usage spike even faster and ultimately failing

5) the sole purpose of this machine/server is to build tiles, nothing else runs on this machine, we did not want anything interrupting this process or bogging down the machine to slow the process.

note: i really wanted to separate the vector data from the raster data, i feel that the sheer size of all the data due to them being combined is an issue in itself. the raster data only changes once a year, whereas the vector data changes weekly. the reasoning behind combining them is so we would not have to make two separate requests for the vector and raster data, thus bogging down the bandwidth of the system.

dave

Re: 10.5.1 Memory Usage

DaveTenney — Wed, 22 May 2019 19:09:07 GMT

I wanted to share more info with everyone and hopefully shed some more light on why this is very perplexing...

i have two images attached...

1) AOI - this is running the "managemapservercachetiles" on level 19 whit an area of interest polygon

a. there is very little work to be done with recreating all tiles within the area of interest.

2) ALL - this running the "managemapservercachetiles" on level 18 and level 19

a. there is a lot of work to be doing recreating all tiles in these two levels

AOI - memory is running up like crazy and has almost maxed out the machine and is about to fail

a. why would this smaller amount of work cause the memory usage?

ALL - the memory usage has stayed consistent at this level throughout the entire process, the cpu usage jumps up and down but i have always seen that behavior.

what in the world is going on?

Re: 10.5.1 Memory Usage

MichaelVolz — Wed, 22 May 2019 19:17:02 GMT

Can you provide a screenshot of either the tool with the parameters filled in or the python call to ManageMapServerCacheTiles_server with all of your parameters?

It's been a long time since I had this process running in 10.2, but I thought the parameters for this call changed between 10.2 and 10.5.1 which could be part of the problem you are seeing.

Re: 10.5.1 Memory Usage

DaveTenney — Wed, 22 May 2019 19:23:20 GMT

here is the screenshot of the tool:

Re: 10.5.1 Memory Usage

DaveTenney — Wed, 22 May 2019 19:25:15 GMT

not sure why it showed up this way, but when we run the tool we leave "wait for job completion" checked, I believe that is the default. again, not sure why in this screenshot it is not checked.

Re: 10.5.1 Memory Usage

MichaelVolz — Wed, 22 May 2019 19:35:29 GMT

Can you take a screenshot of the results showing Inputs from a cache update that would not take that long just to get an idea of what that looks like?

Have you made sure that the AOI geometry in the delta.gdb is reasonable and your script to create the AOI is not creating an area much larger than what you are expecting?

Re: 10.5.1 Memory Usage

DaveTenney — Wed, 22 May 2019 19:45:49 GMT

here are the inputs as seen from geoprocessing results.

Re: 10.5.1 Memory Usage

DaveTenney — Mon, 03 Jun 2019 01:49:23 GMT

all,

would installing and utilizing background geoprocessing assist in memory usage?

I wasn't sure if the background geoprocessing was utilized for the ManageMapServerCacheTiles gptool

thoughts?

dave