Memory Usage on ArcGIS Server increased after upgrade

[SharonGin2]

I had a client who upgraded from 10.3.1 to 10.5.1 few months ago and they reported that the memory usage on the server has increased. There has not been any change in the server configuration or number of services on the server and they are running their servers in a virtualized environment. I believe that they are also having multiple clusters within the site and thus not utilizing the single cluster mode option. Is this expected behaviour after the upgrade?

[DanielUrbach]

How much of an increase are they seeing?

ArcGIS Server 10.5 has some new System services that run that could be adding to the memory usage, although it should be a negligible amount.

[SharonGin2]

they are seeing between 10-25Gb increase in memory usage. Their configuration is such that there are multiple clusters within the site, 1 for map services and another for GP services. Will such configuration cause increase in memory usage since ArcGIS server has moved away from multiple clusters in 10.5.1?

[DannyKrouk]

10-25GB additional memory utilization is a lot.  Is that *per machine*?  Or, aggregated across all machines (in all clusters) in the Site?  I cannot imagine a release-based change that would lead to that magnitude of difference in memory consumption.  Something else is going on.

Did they see this same pattern in pre-production when they did the upgrade there?

[SharonGin2]

10-25Gb increase is per machine. So on 1 machine, it could be 10Gb, and another machine is seeing 25Gb increase in memory. They did mention that they increase the SOC and App Server heap size though, but they did not mention if it was done before or after the upgrade.

[DannyKrouk]

Increasing the SOC heap size is the most likely suspect for that kind of memory growth, based on what you've described.

Increasing the heap size delays Java garbage collection.  That can be necessary in some special circumstances.  But, the need is not very common.  And, when it occurs, it is unlikely to be needed for all services.  Ideally, services that need the larger SOC heap size should be segregated to their own machines (with an appropriate heap size on those machines). The other services should be left on machines with the default heap size.

What reasoning led them to increase the heap sizes (for the App Server and SOC heaps)?  What are the new heap size values?  And, how did they determine that the heap sizes are not larger than necessary?  (Larger heap sizes must use more memory.  Larger-than-necessary heap sizes use more memory than is necessary.  So, the question is whether they are using the memory that they should have been all along or if they have caused themselves to become memory inefficient.)

[SharonGin2]

It turns out that the heap size was already increased in 10.3.1, but memory usage was still acceptable then. After upgrading to 10.5.1, there has not been any change in the heap size but memory usage has surged. They had to reduce the heap size in order to keep the memory usage to a sustainable level. Is there anything in the 10.5.1 version that would adversely impact the memory utilization when heap size is not the default?

[DannyKrouk]

I worry about the changes to the heap size.  The heap sizes should not be increased without specific evidence (from the logs) of heap size shortage.  In addition to being too small, heap sizes can be too large.  The expression of a too large Java heap size is *episodic* excessive memory consumption and performance problems.  So, it is possible that they had the same memory shortage at 10.3.1 but did not notice.  In any event, they should be sure that they've identified the correct heap size for the way they are using ArcGIS Server and stick to it.  If they adjust it in response to memory shortages, they will just move problems around ... not fix them.

I do not know of any specific mechanisms in the 10.5.1 software that would lead to a change in memory consumption on this scale, relative to 10.3.1.

I've elsewhere suggested that they account for the memory utilization so that we know which processes are responsible for that enormous growth.  Is it the App Server, all ArcSOC's, some ArcSOC's, or something else?  Once we know the specific processes that are responsible for the delta, we can start to build specific hypotheses about the cause(s).

[TOPOGRAPHICS]

We seem to have similiar problem with a customer that wants to change from server 10.3 to 10.5.1.

After installing Server 10.5.1 on a new virtual machine the performance of dynamic map-services has decreased.

The datastore (fgdb's) and published map services are exactly the same. Yet on the old server (also a vm) images from map-services are created twice as fast.

We tested both vm's perfomance with the linpac test and microsofts diskspd tool. The new vm performed significantly better in all tests.

The only difference between the servers is the operating system. The old one uses Win Server 2012 and the new one Server 2016. Otherwise the servers are quite the same. No additional software like databases, plenty of unused ram and pagefiles in the 8GB range.

Heap Sizes for ArcGIS Server is 256 MB in both cases.

Is there something special to the 10.5.1 version in regards to performance limiting factors as opposed to the 10.3 version?

[Anonymous User]

I also saw this. We went from 10.2.2 to 10.5.1, and then 10.6.1.  Same gdb's and mxd's.  Same Amazon machine configuration.  2 CPU and 8 gig RAM. It only runs a handful of services and each one is only 200 meg RAM. Literally for now it's only running one website. Same as with you TOPO GRAPHICS, we did go from Win 2012 R2 to 2016. 

We saw a serious decrease in performance and increase in CPU and RAM usage. On 10.5 and 10.6.x. I'm investigating but thus far this is concerning. I have turned off anything that wasn't needed. Even turned off SQL Server (wasn't needed but was on old instance).  So in fact it should run faster, there's a few less things on it.