topic Re: Python Topo To Raster: Tiled Processing in Python Questions

Python Topo To Raster: Tiled Processing

PeterWilson — Thu, 01 Sep 2016 09:41:13 GMT

I'm looking for a reasonable workflow to generate DEM's from contours for multiple study areas without having to clip my contours into tiles in order to generate a DEM. I'm looking for suggestions from the community in being able to process large datasets (contours) by processing the study area into usable tiled areas such as the Tiled processing that ESRI has developed for certain geoprocessing tools:

I'd like to generate a tiled index on the fly based on the extent of my study area and output cellsize, in order to process my study area in tiles and then stitch the tiles together to generate a single DEM.

Topo to Raster is written in Fortran and doesn't take advantage of multiple processors or cores. Its limited in the amount of memory that is allocated to the single geoprocessing tool, so I'm trying to develop a Python workflow of splitting the study area into small enough areas that can be run simultanously and then stitched back together afterwards.

My current study area is made up of nine local municipalities and from my calculations based on a 10m cellsize each local municipality will have the following amount of cells for each DEM:

Baviaans 226 791 600
Blue Crane Route 209 397 384
Camdeboo 250 726 124
Ikwezi 86 625 374
Kouga 50 539 474
KouKamma 73 776 259
Makana 40 357 026
Ndlambe 40 357 026
Nelson Mandela Bay 34 654 290
Sunday's River Valley 114 374 190

Nine Local Municipalities

Re: Python Topo To Raster: Tiled Processing

DanPatterson_Retired — Thu, 01 Sep 2016 10:39:56 GMT

Peter... and you still have the added limitation of using python 2.7 and ArcMap correct?

suggestions from the help (for others in case they haven't seen it

There are insufficient system resources available. The algorithms used in Topo to Raster hold as much information as possible in memory during processing. This allows point, contour, sink, stream, and lake data to be accessed simultaneously.

To facilitate processing of large datasets, it is recommended that unnecessary applications be closed before running the tool to free up physical RAM. It is also important to have sufficient amounts of system swap space on disk. The contour or point input may be too dense for the output cell size specified. If one output cell covers several input contours or points, the algorithm may not be able to ascertain a value for that cell. To resolve this, try any of the following:Decrease the cell size, then resample back to the larger cell size after Topo to Raster.

Rasterize smaller sections of the input data using the Output extent and Margin in cells. Assemble the resulting component rasters with the Mosaic tool. Clip the input data into overlapping sections, and run Topo to Raster separately on each section. Assemble the resulting component rasters with the Mosaic tool.

Use Dice then try Topo To Raster by File since in theory, you should be able to slice and dice, copy and paste the inputs with making minimal changes then run it.

Of course you are still going to be limited by the memory that ArcMap allows

Re: Python Topo To Raster: Tiled Processing

PeterWilson — Thu, 01 Sep 2016 12:14:58 GMT

Hi Dan

Thanks for the following Is this still true if you are using 64 bit background processing and running the python script as a standalone process?

Re: Python Topo To Raster: Tiled Processing

DanPatterson_Retired — Thu, 01 Sep 2016 12:23:22 GMT

Peter... I have just never found the need for the background stuff... perhaps it is the leisurely lifestyle I lead... give it shot on a small sample and time it. (Besides there is also the fact that I have anywhere up to 50 computers I can commandeer should the need arise and I haven't used python 2.7 in quite some time).

So I will leave that for others to comment on

Re: Python Topo To Raster: Tiled Processing

PeterWilson — Thu, 01 Sep 2016 12:33:43 GMT

If only I was so lucky , thanks Dan.

Re: Python Topo To Raster: Tiled Processing

GeorgeNewbury — Thu, 01 Sep 2016 19:25:28 GMT

Interesting problem, given that the typical workflow is to create contours from DEMs.

Did you try using the 'Create Tin' tool? And then make the raster from a TIN?

Re: Python Topo To Raster: Tiled Processing

DanPatterson_Retired — Thu, 01 Sep 2016 19:47:11 GMT

The algorithm behind topo to raster does a pretty good job (anudem) of generating the dem when approaches like you propose fail. There is quite extensive literature and testing behind the implementation and it is generally the preferred method.