Multi-threaded Topo to Raster

02-04-2018 10:07 AM
Occasional Contributor III

I currently still using ArcGIS 10.3 and had to generate a 1m resolution DTM for the whole of Dar es Salaam for a Flood Risk Analysis task on one of our projects. The first problem is that the source of our input data was contour shapefiles at a 1m resolution. I initially created a 5m resolution DTM from the contour datasets for preliminary results which took almost 5 days to run. This was a problem as I had to get the resolution down to 1m to run my analysis. I generated a overlapping index grid allowing for 20% overlap and sliced the contour datasets into 500 tiles. I then generated a Python module utilising Python's Multiprocessing module to process the tiles across 16 cores on our servers. The DTM tiles were completed in under 12 hrs for the entire study area. The Python module which I've attached is below, is my first draft of utilizing Python's Multiprocessing. I'm currently re-writing it now that I have a better understanding how the Multiprocessing module works.

NB: I'd like to thank Duncan Hornby for his post: Create a script tool that uses multiprocessing, that helped me create the following script

Hope the following helps anyone else trying to achieve the same for generating large DTM's

Dar es Salaam: 1m DTM Tiles

Created on 01 Feb 2018


Topo to Raster (3D)

@author: PeterW
# import site-packages and modules
import multiprocessing
import argparse
import arcpy
from functools import partial

# set environment settings
arcpy.env.overwriteOutput = True

# check-out extensions

def dtm_work(tiles_directory, cont):
    Topo to Rasster
    for multiple
    index_name = cont[0]
    inContours = cont[1]
        output_dtm = '{0}\\{1}'.format(tiles_directory, index_name)
        return True
    except Exception as e:
        print e.message
        return False

def dtm_multi(index_grid, tiles_directory):
    Create overlapping
    DTM tiles from
    contour tiles &
    index grid
        cont_param = []
        with arcpy.da.SearchCursor(index_grid, ['SHAPE@', 'PageName', 'Path']) as scur:  # @UndefinedVariable
            for row in scur:
                index_name = row[1]
                input_cont = '{0} {1} {2}'.format(row[2], 'ELEVATION', 'Contour')
                cont_param.append([index_name, input_cont])
        func = partial(dtm_work, tiles_directory)
        arcpy.AddMessage('Sending to pool')
        # declare the number of cores to use, use 2 less than the max
        cpu_num = multiprocessing.cpu_count()-2
        # create the pool object
        pool = multiprocessing.Pool(processes=cpu_num)
        # fire off list to worker function
        # res is a list that is created with what ever the worker function returns
        res =, cont_param)
        # if an error has occurred report it
        if False in res:
            arcpy.AddError('A process\thread failed!')
        arcpy.AddMessage('Finished Multiprocessing DTM Tiles')
    except arcpy.ExecuteError:
        # Geoprocessor through error

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Create overlapping DTM tiles for mosaic')
    parser.add_argument('--index_grid', metavar='path', required=True,
                        help='path to input overlapping index grid feature class')
    parser.add_argument('--tiles_directory', metavar='path', required=True,
                        help='path to output DTM tiles directory')
    args = parser.parse_args()
0 Kudos
1 Reply
MVP Legendary Contributor

Nice Peter... going to try this out.

0 Kudos