Higher RAM usage for services with SOE enabled

[EvelynHsu]

If a service has several SOE enabled, does its ArcSOC process then consume more RAM as compared to having no SOE enabled?

One of our clients uses ArcGIS Server 10.8.1 and the site has over 150 services. On top of these, each service often has 3-4 SOE enabled. The site crashes frequently due to OS error "out of memory". We suggested to migrate some of the services to a new site, reduce the number of services in each site, and also reduce the min instances of each service (or use Shared Instances). The large number of SOEs is a huge concern too, but there isn't clear documentation about whether SOE causes a higher usage of RAM. 

Any idea?

[Scott_Tansley]

150 services is quite high.  Are they all dedicated or are using the sharing model?  There are many factors that determine memory usage, without even considering SOEs.  Out of interest how much RAM is available on each host?

[EvelynHsu]

Each machine was allocated 32GB RAM.

All services were published with ArcMap, currently using dedicated instances. 

[Scott_Tansley]

Without doing a full capacity plan, I’d be looking at 48-64 GB, but preferably no more than 100 services per site.  Also have a play with this tool:  https://www.ope.nz/esri/mem_calc_advanced_104 

[Scott_Tansley]

The other alternative is publishing with pro and shared instances, they really make a massive difference to memory usage.

please be aware changes in the code may be required:  https://developers.arcgis.com/enterprise-sdk/guide/net/develop-extensions-for-shared-instances-net/ 

[EvelynHsu]

Thanks Scott. That's a really useful tool!

We were under the impression that if a service has SOE/SOI, it can't be directly converted to Shared Instances, since the SOE/SOI will need to be re-developed using Pro SDK. This is also why we decided to stay with Dedicated Instances, and maybe try with lowering Min Instances to 0 for those rarely used services.

[Scott_Tansley]

Instances to zero is an often used approach.  The service will remain on but will need time to spin up when first requested.  If I used it will close down and then the next user needs the spin up time.   

it’s an approach, but also a compromise of performance.  Given the need to use pro in the future also consider that at 11.x you need to use .Net 6 which means more change.  

There’s a few things to consider and some careful planning needed for the road ahead.