How much does merging services affect server performance?

[YanPustynnyy]

Hello!0

Currently we have an ArcGIS Server 10.4 up and running with more than 150 separate services which affects the server performance really bad. RAM usage is almost always very high as well as the number of ArcSOC processes and swap file is large. Since there are a lot of services related to a common topic (f.i. climate, geology etc) I was thinking of merging some of them to improve performance but really have little idea how effective it will be. Will there be a somewhat significant improvements in RAM usage and overall server performance if I there are, for example, 50 services instead of 100 but with twice more layers? Thank you

Yan

[JonathanQuinn]

It's tough to say that adding X layers will change RAM consumption by Y as there are too many variables to give a concrete answer. I don't think it's linear, though. For example, if a service with 50 layers uses 100 MB, a service with 100 layers shouldn't use 200 MB but I've never tested or monitored that. Best approach is to test and see.

You could also look into setting some of them to have 0 min instances. A SOC won't be started unless a request comes in for the service, but there will be an initialization cost as it spins one up when the first request arrives.

[YanPustynnyy]

Thanks for the response!

I made some tests related to this topic and it seems that merging services actually drops RAM usage a bit. I merged 5 services into 1 and RAM usage decreased by about 5%. I guess the affect would be larger if more services is merged. 
I have also set min instances to 0 of almost all secondary services and RAM usage dropped significatly but it's going up anyway if the user open maps containing them I guess. 
I will be investigating it further and try to lower RAM consumption as possible when the server is heavily loaded.

Yan

[JoshuaDalton]

A few thoughts as I've been doing a lot of memory consumption firefighting and performance tweaking on our arcgis server apps...

Merging services has minimal impact, but 5% is 5% so I guess you should take it, but note that grouped layers don't work with all widgets in webappbuilder (like, the screening widget) so check what your usage goals are first.

Adjusting machine level java limits and step sizing can have considerable impacts on memory consumption.  Consider looking into the Xmx and Xms settings (Xmx can be set in arcgis/admin - machines - <machine name> for the server java process as App Heap, or your individual services as SOC Heaps, which show up as parameters in the java and ArcSOC command lines if you look in task manager after adding the command line column.

You can use the windows environment variable for _JAVA_OPTIONS= ...... overriding settings here ...... to add command line settings for the whole machine.

Of additional interest, which I've yet to try, is the -XX:MaxHeapFreeRatio which must be used in combo with -XX:+UseG1GC or -XX:+UseSerialGC which may allow heap to shrink after garbage collection.  I've NOT tried this, and barely came across it today, so do research it first.

Regarding performance, certainly increasing the app heap and the SOC heap in the admin console helped improve performance considerably, and we've begun setting the app to 1024MB on several of our servers as the reduction in garbage collection cpu time made a huge performance impact.  Be warned though that adjusting SOC sizing will quickly use up all Virtual memory by spiking commit charge.  Look under Details tab on windows server Task Manager and add commit size to see what really goes into using up that committed memory total.  Note that exceeding the max committed memory will cause exceptions and all kinds of instability.  If you blow way past it, you either need to ramp up your virtual memory for long enough to start ags and turn the arcsoc setting down, or get into the admin console and turn it down before the ArcSOCs launch.

[MarkChilcott]

We have just had a difficult year in relation to server crashes and performance tuning.  Following are the best references I could give you are:

• ArcGIS Enterprise: Performance and scalability best practices
• Server Software Performance
• PerfQAnalyzer for ArcMap
• System Log Parser for ArcGIS
• Shared instances in 10.7

Some hints from our experience:

• Use the System Log Parser to work out which of your services are consuming the most resources.  Focus on the top 20.
• Tune the number of instances first.  Pay a lot of attention to this
• Tune the individual mxd files to ensure these are working to the best ability.  
• Use ArcGIS monitor or something like Grafana to monitor the hardware
• Get the the latest release of software as soon as you can
• Be very careful of the ArcGIS Server Unintended Service Restart bug - make sure you install the patch for this

You may find shared instances in 10.7 has some advantages.

Cheers,

Mark

[MichaelVolz]

Have you found AGS 10.7 shared instances helpful to drop memory consumption on your server?

[MarkChilcott]

We moved from 10.3.1 to 10.6.1.  10.7.1 was not released at the time we started the migration.  During our tuning we wanted to reduce load on servers, and set min instances to 0.  Had the cold start up issue though.  Is a balancing act.  Hoping that 10.7.1 shared instances will get around this problem, but have not tested.

Here is another hint - don't leave it too long between upgrades.  Do it often.  A four year gap is a killer.