DataStore and RAM issue

[VHolubec]

Dear all,

I have a question to Data Store RAM usage. I have 10.8 Enterprise (all in one machine), with 7 services (3 features, 2 map service and 2 vector tile layers services) and 5 others non-hosted services.

The problem is with RAM usage of Open JDK Process. It takes more than 21 GB of RAM, so it´s huge consumption against other Server and Portal Processes.

It´s true that all 3 DataStores are enabled: Relational, Tile Cache, Spatiotemporal. But there is no Tile service or service which could be using spatiotemporal data store, so I am quite confuse, what could cause such huge consumption of the RAM.

The restart of the process helps for a while, but with more time running the Data Store use more and more RAM.

Does anyone have faced similar issue?  Philip Heede don´t you know if this is just a common state of art, or something what could be optimized by settings in our DataStore?

Thank you.

Vladimir

[ElliotJones · Accepted solution]

Hi Vladimír Holubec‌,

Without getting into the content of your services, based on the description of your deployment, it's very likely that you're overtaxing your machine's resources by placing all three Data Store types on the same machine. The software requirements recommend 8GB RAM each for both relational and tile cache, and an additional 16GB minimum for the spatiotemporal data store (which is recommended to be run on its own Data Store machine). When you throw in the RAM requirements for Portal for ArcGIS and ArcGIS Server (8GB each, more for ArcGIS Server with certain licensing roles), there's a high likelihood for excessive resource competition on the machine.

Relational and tile cache data stores preallocate memory so that it's available when needed, and the spatiotemporal data store takes up approximately 40% of available RAM, which might explain the steady rise in RAM consumption over time.

[VHolubec]

Dear Elliot Jones‌

thank you very much for response. I did un-registration of spatiotemporal data Store via CMD utility, then deleted all hosted FS, restarted server, portal datastore, but the situation is still the same. 

Process - C:\Program Files\ArcGIS\DataStore\framework\runtime\jre\bin, takes still more than 20 GB of RAM.

Do I still miss something? The machine has 45 GB of RAM, so just one JAVA DataStore process takes almost 50% of RAM, it seems to me too much.

Thank you.

Regards,

Vladimir

[JonathanQuinn]

Can you post the full command line arguments for the process? Is it the javaw.exe process? Does it immediately or gradually climb to 20 GB?

[VHolubec]

Dear Jonathan Quinn‌,

the command line pointing to elastic search component. 

"C:\Program Files\ArcGIS\DataStore\framework\runtime\jre\bin\java.exe"  -Xms23173m -Xmx23173m -XX:CMSInitiatingOccupancyFraction=75 -XX:-UseConcMarkSweepGC -XX:-UseCMSInitiatingOccupancyOnly -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=75 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=C:/arcgis/arcgisdatastore/temp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Djava.locale.providers=COMPAT -XX:UseAVX=2 -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=C:/arcgis/arcgisdatastore/temp -Dio.netty.allocator.type=pooled -XX:MaxDirectMemorySize=12150898688 -Delasticsearch -Des.path.home="C:\Program Files\ArcGIS\DataStore\framework\runtime\elasticsearch" -Des.path.conf="C:\Program Files\ArcGIS\DataStore\framework\runtime\elasticsearch\config" -Des.distribution.flavor="oss" -Des.distribution.type="zip" -Des.bundled_jdk="true" -cp "C:\Program Files\ArcGIS\DataStore\framework\runtime\elasticsearch\lib\*" "org.elasticsearch.bootstrap.Elasticsearch" -d -p "C:\Program Files\ArcGIS\DataStore\framework\etc\pids\elastic.pid"   

But the spatiotemporal data store is not configured here. The RAM is loaded almost immediately the process has started - see video: ftp://ftp.arcdata.cz/outgoing/holubec/DataStore.mp4

The settings of the DataStore is default. Installation was done via Enterprise Builder. All available patches are installed.

Thank you for any idea.

Vladimir

[RachelSears · Accepted solution]

After unregistering, if the spatiotemporal data store still shows up when running the describedatastore command line utility, you may need to remove the entry for the spatiotemporal data store in the arcgis-data-store-config.json file. The default location is C:\arcgisdatastore\etc\arcgis-data-store-config.json

Remove the entry like the one I have highlighted, save the file, and then re-start the Data Store service.

After waiting a few more minutes, run the describedatastore command line tool again to make sure it is no longer showing an entry for Spatiotemporal.

Hope this helps!

- Rachel

[VHolubec]

Dear Rachel Sears‌,

the machine does not have any external AVG software installed. I made the manual editing of  the config.file, and this seems it solve the issue. 

Thank you very much!

Vladimir

[WarrenMedernach]

We are running into a very similar situation at the moment.

We had our Enterprise running on an Azure VM with 56GB of ram.  This implementation had Portal, Server, and the data stores all on one machine, and a secondary machine for an Image Server.  This system ran great as configured for almost 2yrs with no issues, and the memory usage hovered around 50-60% consistently. 

Due to unforeseen circumstances we needed to migrate the entire system to a new VM.  We built a new VM based on the exact same system specs as the previous system.  Ever since we deployed this new system there is a java process that consumes 30GB of memory within minutes of starting the system and the memory consumption sits at +92-96% and never drops.  Needless to say our current system is almost inoperable.

Windows Defender is the only anti-virus software running and all appropriate software and system folders have been added to the Defender exception list.

Any ideas what might be causing this?

[RachelSears]

Do you have an antivirus software? You may need to add exceptions for/whitelist the following directories:

• Portal for ArcGIS
  • <installation drive>:\arcgisportal
  • <installation drive>:\Program Files\ArcGIS\Portal
• ArcGIS Server
  • <installation drive>:\arcgisserver
  • <installation drive>:\Program Files\ArcGIS\Server
  • <installation drive>:\Python27
• ArcGIS Data Store
  • <installation drive>:\arcgisdatastore
  • <installation drive>:\Program Files\ArcGIS\Datastore