Elevation cache calculation - slow performance

[MarkMindlin]

Hi,

We have very heavy elevation data >10 terabytes of *.asc files of 1 or 2 or 5 metres.

• amount of ascii files about 40,000
• ascii files organized in subfolders
• size of an individual ascii file from 1 MB to 40MB.

The caching machine that we run 21 cores (from 24) generates only about one Gbyte cache per hour. It has been running more than two week as far.

We are looking to ways how to increase performance.

1. transform *.asc files before adding to Mosaic Dataset 

2. calculate overviews before calculating a cache

3. any other solution/ideas

Geoprocessing tool "Manage Tile Cache" (NOT ON THE SERVER), ArcGIS Desktop 10.5

Any help please.

[DuncanHornby]

Amend your question to say:

• how many ascii files you are talking about
• how they are organized (i.e. in subfolders or dumped in one folder?)
• the size of an individual ascii file KB or MB?
• row and column numbers
• state type (integer or float32/64 bit)?

[MarkMindlin]

Thanks Dunkan,

I have amended amount, subfolders, size.

How I see

• row and column numbers
• state type (integer or float32/64 bit)?

[MarkMindlin]

Probably, we are not correct to select way we prepare data for publishing. What is the best way to prepare and publish?

ESRI help says:

Prepare source data

Obtain source elevation data for your area of interest. The source data can be from LIDAR, photogrammetry, spot, or contour sources, but you need to create a DTM from it. Save the derived raster DTM as a stand-alone TIFF or load it into a geodatabase as a raster dataset

[DuncanHornby]

I have to admit I've never used the Manage Tile Cache tool, but reading the help it does state in the usage section:

This tool may take a long time to run for caches that cover a large geographic extent or very large scales.

 A 1m resolution dataset is pretty large...

So your source data is as ASCII format but the input into the cache tool is a Mosaic dataset? I wonder if there is some performance hit converting the ASCII into a raster (behind the scenes) before it is cached? ASCII is just a text file...

Also tool honors the environment setting parallel processing factor, have you set that to make best use of your cores?

If it were me, I would try with a sample of your data as is then compare the processing speed of that to a mosaic dataset where the source rasters are tiff files, so you have already converted your ascii to tiffs as a separate pre-processing step.

Just an idea? May be someone else can suggest something and this is not territory I usually work in.

Good luck!

[MarkMindlin]

Thanks Duncan,

Appreciate your help.

Yes, we use parallel processing factor otherwise how we run 21 from 24 cores

Yes, I am thinking the same direction of pre-processing to TIF