Hosted scene layers takes forever to cache on server

[Magdalena]

Hi,

We are struggeling with long caching times on hosted scene layers in ArcGIS Enterprise 11.5.

We are working on our BIM-GIS-integration and want to be able to visualize some of our BIM data in GIS. We are using Python and want to visualize up-to-date BIM-data in a 3D scene in ArcGIS Enterprise.

To be able to filter and query the data in the 3D scene, we must have the associated feature layer. These means we can't create a local cache and publish this, we have to use the scenelayersharingdraft and create the cache on server (hopefully this will change in future versions).

A show-stopper for us right now is that caching of the data can take more than 24 hours - meaning the data shown in GIS is not up-to-date compared to the data on the BIM platform.

We've spent a lot of time testing, and want to know if the behaviour we are seeing is normal and expected or if there is something we could do to improve the caching performance.

We are dealing with complex data - we know that. One of our tests has been performed with 7 IFC-files as input. These are merged into 1 multipatch file. This file contains around 44 mill vertices, and around 5000 multipatch objects. When trying to publish this (and create the cache on the server) we experienced that it took over 24 hours to cache. To further test these data we created 7 new multipatch files, and in each of them we removed all objects related to one IFC-file. All power lines were removed in one file and all roads removed in another, etc.

As a test, we attempted to create a local cache based on these 7 new multipatch files, and this takes time as well. The fastest of these 7 files took around 1 hour to cache locally, the other 6 files took between 8-15h to cache. The "fastest" file is the multipatch file in which the objects from the biggest IFC-file were removed (2 objects and around 26 mill vertices). When I try to create a cache of this file on the server, it takes around 2,5 hours - so more than twice the time compared to a local cache. Also, if I publish the 2 objects with 26 mill vertices from the biggest IFC-file separately, it takes around 2 minutes to cache on the server. The 6 other multipatch files that includes this file takes around 8-15h to cache, meaning 7-14 hours more because of a file that takes 2 minutes to cache. We've heard it can be an issue to cache complex structures that are within another structure. Therefore I took the most complex file and merged it with the building object it is inside of (68 objects and a bit more than 26 mill vertices). This took around 2 minutes to cache as well.

Resources on the server seems OK to us. CPU and RAM is far below 100%. The only things that bugs me a bit is that when I create a local cache, I can see the process often uses around 80-110 GB RAM, on the server it looks like it's around 20GB RAM used. Is it not able to use all available resources?

So local caching is also slow, meaning these data are complex and it is not only a server issue. We know that we can simplify the data but as for now we want to know is this behavior is normal and expected or if there is something we could do. Is the processing improved in future versions of Enterprise? If it's taking twice the time with cache building on the server (or maybe even more when the data is more complex?) we have to significantly lower our expectations of this BIM-GIS integration.

This is what we have checked on the server side:
- Clean install of ObjectStore. Main program on C:\Program Files , staging catalog on D:\, log catalog on D:\
- Port verification: Used Test-NetConnection in PowerShell to make sure all neccessary ports were open.
Test-NetConnection -ComputerName ObjectStore -Port 29879
Test-NetConnection -ComputerName ObjectStore -Port 19879
Test-NetConnection -ComputerName ObjectStore -Port 2443
Test-NetConnection -ComputerName ObjectStore -Port 6443
Test-NetConnection -ComputerName ObjectStore -Port 11211
They all returned TcpTestSucceeded: True
- Increased RAM on ObjectStore VM from 64 GB to 128 GB.
- Checked network traffic between ObjectStore and Hosting:

Direction, Total traffic, Rate, Port

1	Hosting → Object DS	646,1 MB	62,7 Kbps	29879
2	Object DS → Hosting	12,7 MB	1,2 Kbps	6443
3	Hosting → Object DS	148,9 KB	14,1 bps	2443

Important information about our infrastructure:
- ArcGIS Enterprise 11.5 with Object Store
- Single node without cluster
- Hosting server details: Version 11.5.0 (build 56755), Hardware 24 CPU Physical, 24 CPU Logical, 64 GB RAM, OS Microsoft Windows Server 2022 build 20348
- ObjectStore details: Version 11.5.0 (build 56755), Hardware 16 CPU Physical, 16 CPU Logical, 128 GB RAM, OS Microsoft Windows Server 2022 build 20348

Any ideas? Thanks!

[GlenterpriseUK]

Hi @Magdalena,

From what you are describing and the testing carried out on your side, the challenge you are facing seems related to the complexity of BIM geometry/data itself rather than a machine or ArcGIS Enterprise related.

Since local cache generation is taking many hours, I would focus on a specific test case to reduce the complexity of the source data.

Example: Removing unnecessary internal elements (such as detailed MEP components), Reducing vertex counts where possible and splitting the data into smaller, more manageable layers.

I am sure that testing with an optimised subset will show differences compared to what you have already observed.

If querying and filtering capabilities are required, another option would be to keep a lightweight feature layer for attributes while using a simplified scene layer for visualisation only.

I would not expect a significant improvement from adding more hardware resources.

Regards,

Glen

[Magdalena]

Hi, thanks a lot for your reply.

I agree that the complexity of the data is a thing here, and our testing has shown that caching time can be significantly reduced with reduced data complexity.

Though, I want to understand what it is that makes these data so problematic to cache. This to be able to avoid these situations in the future since we're trying to automate this integration. I find it very hard to understand this part (from main post):

"Also, if I publish the 2 objects with 26 mill vertices from the biggest IFC-file separately, it takes around 2 minutes to cache on the server. The 6 other multipatch files that includes this file takes around 8-15h to cache, meaning 7-14 hours more because of a file that takes 2 minutes to cache. We've heard it can be an issue to cache complex structures that are within another structure. Therefore I took the most complex file and merged it with the building object it is inside of (68 objects and a bit more than 26 mill vertices). This took around 2 minutes to cache as well."

The caching time is not linear with the amount of data. There's something else here that I don't understand.

Also, is it normal that it takes more than twice the time to cache the data on server compared to locally?

Thanks!

[M678]

Ran a few tests with scene layers and what seemed to work best was uploading locally generated slpk's and let Portal handle creating the service.

Would it be possible to share that mega 24h file somewhere? Could be a good testcase.

Regarding cpu: most proceses are limited by its single-core speed. Servers usually run Xeons while your pc/laptop probably has a cpu with less cores, but around twice the core speed (in transactions/second). So it makes sense for locally run tools to run about twice as fast vis-a-vis the server. That is, provided you have sufficient spare ram available.

As for the ram; the object store  (and spatiotemporal) uses a dynamic formula:

changedbproperties 

With the '--heap-size' option you can change it to a fixed value. Think I set it to 23456 or something; prefer to have it static since it's a single-machine environment.