Multi-threaded Topo to Raster
I currently still using ArcGIS 10.3 and had to generate a 1m resolution DTM for the whole of Dar es Salaam for a Flood Risk Analysis task on one of our projects. The first problem is that the source of our input data was contour shapefiles at a 1m resolution. I initially created a 5m resolution DTM from the contour datasets for preliminary results which took almost 5 days to run. This was a problem as I had to get the resolution down to 1m to run my analysis. I generated a overlapping index grid allowing for 20% overlap and sliced the contour datasets into 500 tiles. I then generated a Python module utilising Python's Multiprocessing module to process the tiles across 16 cores on our servers. The DTM tiles were completed in under 12 hrs for the entire study area. The Python module which I've attached is below, is my first draft of utilizing Python's Multiprocessing. I'm currently re-writing it now that I have a better understanding how the Multiprocessing module works.
NB: I'd like to thank Duncan Hornby for his post: Create a script tool that uses multiprocessing, that helped me create the following script 
Hope the following helps anyone else trying to achieve the same for generating large DTM's
Dar es Salaam: 1m DTM Tiles
'''
Created on 01 Feb 2018
Multi-threaded
Topo to Raster (3D)
@author: PeterW
'''
# import site-packages and modules
import multiprocessing
import argparse
import arcpy
from functools import partial
# set environment settings
arcpy.env.overwriteOutput = True
# check-out extensions
arcpy.CheckOutExtension('3D')
def dtm_work(tiles_directory, cont):
"""
Multiprocess
Topo to Rasster
for multiple
DTMs
"""
index_name = cont[0]
inContours = cont[1]
try:
output_dtm = '{0}\\{1}'.format(tiles_directory, index_name)
arcpy.TopoToRaster_3d(inContours,
out_surface_raster=output_dtm,
cell_size=1)
return True
except Exception as e:
print e.message
return False
def dtm_multi(index_grid, tiles_directory):
"""
Create overlapping
DTM tiles from
contour tiles &
index grid
"""
try:
cont_param = []
with arcpy.da.SearchCursor(index_grid, ['SHAPE@', 'PageName', 'Path']) as scur: # @UndefinedVariable
for row in scur:
index_name = row[1]
input_cont = '{0} {1} {2}'.format(row[2], 'ELEVATION', 'Contour')
cont_param.append([index_name, input_cont])
func = partial(dtm_work, tiles_directory)
arcpy.AddMessage('Sending to pool')
# declare the number of cores to use, use 2 less than the max
cpu_num = multiprocessing.cpu_count()-2
# create the pool object
pool = multiprocessing.Pool(processes=cpu_num)
# fire off list to worker function
# res is a list that is created with what ever the worker function returns
res = pool.map(func, cont_param)
pool.close()
pool.join()
# if an error has occurred report it
if False in res:
arcpy.AddError('A process\thread failed!')
arcpy.AddMessage('Finished Multiprocessing DTM Tiles')
except arcpy.ExecuteError:
# Geoprocessor through error
arcpy.AddError(arcpy.GetMessages(2))
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Create overlapping DTM tiles for mosaic')
parser.add_argument('--index_grid', metavar='path', required=True,
help='path to input overlapping index grid feature class')
parser.add_argument('--tiles_directory', metavar='path', required=True,
help='path to output DTM tiles directory')
args = parser.parse_args()
dtm_multi(index_grid=args.index_grid,
tiles_directory=args.tiles_directory)
