topic Re: Elevation cache calculation - slow performance in Geoprocessing Questions

Elevation cache calculation - slow performance

MarkMindlin — Wed, 12 Jul 2017 00:06:27 GMT

Hi,

We have very heavy elevation data >10 terabytes of *.asc files of 1 or 2 or 5 metres.

amount of ascii files about 40,000
ascii files organized in subfolders
size of an individual ascii file from 1 MB to 40MB.

The caching machine that we run 21 cores (from 24) generates only about one Gbyte cache per hour. It has been running more than two week as far.

We are looking to ways how to increase performance.

1. transform *.asc files before adding to Mosaic Dataset

2. calculate overviews before calculating a cache

3. any other solution/ideas

Geoprocessing tool "Manage Tile Cache" (NOT ON THE SERVER), ArcGIS Desktop 10.5

Any help please.

Re: Elevation cache calculation - slow performance

DuncanHornby — Wed, 12 Jul 2017 13:34:38 GMT

Amend your question to say:

how many ascii files you are talking about
how they are organized (i.e. in subfolders or dumped in one folder?)
the size of an individual ascii file KB or MB?
row and column numbers
state type (integer or float32/64 bit)?

Re: Elevation cache calculation - slow performance

MarkMindlin — Wed, 12 Jul 2017 23:43:05 GMT

Thanks Dunkan,

I have amended amount, subfolders, size.

How I see

row and column numbers
state type (integer or float32/64 bit)?

Re: Elevation cache calculation - slow performance

MarkMindlin — Wed, 12 Jul 2017 23:54:00 GMT

Probably, we are not correct to select way we prepare data for publishing. What is the best way to prepare and publish?

ESRI help says:

Prepare source data

Obtain source elevation data for your area of interest. The source data can be from LIDAR, photogrammetry, spot, or contour sources, but you need to create a DTM from it. Save the derived raster DTM as a stand-alone TIFF or load it into a geodatabase as a raster dataset

Re: Elevation cache calculation - slow performance

DuncanHornby — Thu, 13 Jul 2017 09:37:27 GMT

I have to admit I've never used the Manage Tile Cache tool, but reading the help it does state in the usage section:

This tool may take a long time to run for caches that cover a large geographic extent or very large scales.

A 1m resolution dataset is pretty large...

So your source data is as ASCII format but the input into the cache tool is a Mosaic dataset? I wonder if there is some performance hit converting the ASCII into a raster (behind the scenes) before it is cached? ASCII is just a text file...

Also tool honors the environment setting parallel processing factor, have you set that to make best use of your cores?

If it were me, I would try with a sample of your data as is then compare the processing speed of that to a mosaic dataset where the source rasters are tiff files, so you have already converted your ascii to tiffs as a separate pre-processing step.

Just an idea? May be someone else can suggest something and this is not territory I usually work in.

Good luck!

Re: Elevation cache calculation - slow performance

MarkMindlin — Fri, 14 Jul 2017 01:00:38 GMT

Thanks Duncan,

Appreciate your help.

Yes, we use parallel processing factor otherwise how we run 21 from 24 cores

Yes, I am thinking the same direction of pre-processing to TIF