Arc 10 - geoprocessing results - "Succeeded" but still processing @ 80% 13 hrs later

Question

I decided to see about these newfangled raster mosaic datasets, so I took 133 DEMs from 24 bathymetry surveys (bags that were converted to tifs and corrected from MLLW to NAVD88), and followed the instructions in help to create and add them to a mosaic dataset.

The process supposedly  completed (according to geoprocessing results) but the geoprocessing status bar is stuck at 80% (or something like that - it's hard to read since the progress text doesn't actually scroll all the way through, but resets to x0% and immediately scrolls out of the window)

I've included the whole geoprocessing results log as a PDF attachment since the txt file limit is 12KB. Any thoughts?

MosaicDatasetGPResults.pdf

Screen Shot 2021-10-19 at 12.00.46 PM.jpg

KimOllivier · Answer

I didn't confuse the commands, I should have made it clearer that I meant any tool that 'merges rasters'. The Mosaic help does not warn you not to add a lot of tiles at once, and I expect that the mosiac dataset builder has similar limits, probably using the same underlying calls. It does not seem that the new command has solved all the space problems as we might have expected.

Have you tried building the mosaic dataset a few tiles at a time?

AndyRitchie · Answer

I have always found that there are very small practical limits of tiles when using Mosaic. I have much more success by merging strips (or a box) of tiles, then merge the strips, then merge the final set.

Thanks for your reply, Kim, but I think you may have confused the mosaicking process, which I was not posting about, with the new "mosaic dataset" introduced in ArcGIS 10, which should be more robust than the old mosaicking process.

Andy

KimOllivier · Answer

I have always found that there are very small practical limits of tiles when using Mosaic. I have much more success by merging strips (or a box) of tiles, then merge the strips, then merge the final set. It all sounds a bit messy, so that is why I script the process, and it helps if the tile names have a pattern. Max count < 10 at a time. In the end I have a grid of 300 tiles that performs surprisingly well.

For more a more general approach to the "13 hours" I use these strategies:

Apply the "Cup of Coffee Rule". If a single process has not completed in the time to have a cup of coffee, then interrupt the process and find a better way.
 That might be a different tool, or different software even...

I find that nothing of interest results leaving processes running too long, they are corrupt or incomplete because the system has run out of memory, disk space or otherwise and is useless.

My suggestions are: Make sure your PC is tuned: plenty of local disk space, defragged, and a temp workspace defined as a file geodatabase (the default in ArcGIS 10). This ensures you have enough space for temporary files that are really large in raster processing.

Do some calculations on the 'processing effort' if you can. Esri staff suggest you try a sample first, but that does not really tell you if the process will scale for larger sets. So maybe just tile the data and do it in pieces if a small part works in "Coffee Time".

Have a close look at your data. Unexpected data is often the problem, conflicting projections, null values, missmatch of cell sizes, default tolerances, and many more. Check vectors are clean, and that you have all indexes set, use file geodatabases, not shapefiles. Make sure you do not have single large polygons covering the area that you are using for processing. Dice them up, there is a new tool for that.

For mosaic, just calculate how large the target tile will be with 133 tiles and multiply up to see how much memory that will take uncompressed. Probably more than Amazon's servers.

Basically ArcGIS will not warn you are attempting something impossible, so you have to be wary.