DataStore and RAM issue

297
7
Jump to solution
06-25-2020 01:44 AM
Highlighted
Esri Contributor

Dear all,

I have a question to Data Store RAM usage. I have 10.8 Enterprise (all in one machine), with 7 services (3 features, 2 map service and 2 vector tile layers services) and 5 others non-hosted services.

The problem is with RAM usage of Open JDK Process. It takes more than 21 GB of RAM, so it´s huge consumption against other Server and Portal Processes.

It´s true that all 3 DataStores are enabled: Relational, Tile Cache, Spatiotemporal. But there is no Tile service or service which could be using spatiotemporal data store, so I am quite confuse, what could cause such huge consumption of the RAM.

The restart of the process helps for a while, but with more time running the Data Store use more and more RAM.

Does anyone have faced similar issue?  Philip Heede don´t you know if this is just a common state of art, or something what could be optimized by settings in our DataStore?

Thank you.

Vladimir

Reply
0 Kudos
1 Solution

Accepted Solutions
Highlighted
Occasional Contributor II

After unregistering, if the spatiotemporal data store still shows up when running the describedatastore command line utility, you may need to remove the entry for the spatiotemporal data store in the arcgis-data-store-config.json file. The default location is C:\arcgisdatastore\etc\arcgis-data-store-config.json

Remove the entry like the one I have highlighted, save the file, and then re-start the Data Store service.

After waiting a few more minutes, run the describedatastore command line tool again to make sure it is no longer showing an entry for Spatiotemporal.

Hope this helps!

- Rachel

View solution in original post

Reply
0 Kudos
7 Replies
Highlighted
Esri Contributor

Hi Vladimír Holubec‌,

Without getting into the content of your services, based on the description of your deployment, it's very likely that you're overtaxing your machine's resources by placing all three Data Store types on the same machine. The software requirements recommend 8GB RAM each for both relational and tile cache, and an additional 16GB minimum for the spatiotemporal data store (which is recommended to be run on its own Data Store machine). When you throw in the RAM requirements for Portal for ArcGIS and ArcGIS Server (8GB each, more for ArcGIS Server with certain licensing roles), there's a high likelihood for excessive resource competition on the machine.

Relational and tile cache data stores preallocate memory so that it's available when needed, and the spatiotemporal data store takes up approximately 40% of available RAM, which might explain the steady rise in RAM consumption over time.

Highlighted
Esri Contributor

Dear Elliot Jones

 

thank you very much for response. I did un-registration of spatiotemporal data Store via CMD utility, then deleted all hosted FS, restarted server, portal datastore, but the situation is still the same. 

 

Process - C:\Program Files\ArcGIS\DataStore\framework\runtime\jre\bin, takes still more than 20 GB of RAM.

 

Do I still miss something? The machine has 45 GB of RAM, so just one JAVA DataStore process takes almost 50% of RAM, it seems to me too much.

Thank you.

Regards,

Vladimir

Reply
0 Kudos
Highlighted
Esri Frequent Contributor

Can you post the full command line arguments for the process? Is it the javaw.exe process? Does it immediately or gradually climb to 20 GB?

Reply
0 Kudos
Highlighted
Esri Contributor

Dear Jonathan Quinn‌,

the command line pointing to elastic search component. 

"C:\Program Files\ArcGIS\DataStore\framework\runtime\jre\bin\java.exe"  -Xms23173m -Xmx23173m -XX:CMSInitiatingOccupancyFraction=75 -XX:-UseConcMarkSweepGC -XX:-UseCMSInitiatingOccupancyOnly -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=75 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=C:/arcgis/arcgisdatastore/temp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Djava.locale.providers=COMPAT -XX:UseAVX=2 -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=C:/arcgis/arcgisdatastore/temp -Dio.netty.allocator.type=pooled -XX:MaxDirectMemorySize=12150898688 -Delasticsearch -Des.path.home="C:\Program Files\ArcGIS\DataStore\framework\runtime\elasticsearch" -Des.path.conf="C:\Program Files\ArcGIS\DataStore\framework\runtime\elasticsearch\config" -Des.distribution.flavor="oss" -Des.distribution.type="zip" -Des.bundled_jdk="true" -cp "C:\Program Files\ArcGIS\DataStore\framework\runtime\elasticsearch\lib\*" "org.elasticsearch.bootstrap.Elasticsearch" -d -p "C:\Program Files\ArcGIS\DataStore\framework\etc\pids\elastic.pid"   

But the spatiotemporal data store is not configured here. The RAM is loaded almost immediately the process has started - see video: ftp://ftp.arcdata.cz/outgoing/holubec/DataStore.mp4

The settings of the DataStore is default. Installation was done via Enterprise Builder. All available patches are installed.

Thank you for any idea.

Vladimir

Reply
0 Kudos
Highlighted
Occasional Contributor II

After unregistering, if the spatiotemporal data store still shows up when running the describedatastore command line utility, you may need to remove the entry for the spatiotemporal data store in the arcgis-data-store-config.json file. The default location is C:\arcgisdatastore\etc\arcgis-data-store-config.json

Remove the entry like the one I have highlighted, save the file, and then re-start the Data Store service.

After waiting a few more minutes, run the describedatastore command line tool again to make sure it is no longer showing an entry for Spatiotemporal.

Hope this helps!

- Rachel

View solution in original post

Reply
0 Kudos
Highlighted
Esri Contributor

Dear Rachel Sears‌,

the machine does not have any external AVG software installed. I made the manual editing of  the config.file, and this seems it solve the issue. 

Thank you very much!

Vladimir

Reply
0 Kudos
Highlighted
Occasional Contributor II

Do you have an antivirus software? You may need to add exceptions for/whitelist the following directories:

  • Portal for ArcGIS
    • <installation drive>:\arcgisportal
    • <installation drive>:\Program Files\ArcGIS\Portal
  • ArcGIS Server
    • <installation drive>:\arcgisserver
    • <installation drive>:\Program Files\ArcGIS\Server
    • <installation drive>:\Python27
  • ArcGIS Data Store
    • <installation drive>:\arcgisdatastore
    • <installation drive>:\Program Files\ArcGIS\Datastore
Reply
0 Kudos