Multi-threaded Topo to Raster

PeterWilson — Sat, 11 Dec 2021 15:25:33 GMT

I currently still using ArcGIS 10.3 and had to generate a 1m resolution DTM for the whole of Dar es Salaam for a Flood Risk Analysis task on one of our projects. The first problem is that the source of our input data was contour shapefiles at a 1m resolution. I initially created a 5m resolution DTM from the contour datasets for preliminary results which took almost 5 days to run. This was a problem as I had to get the resolution down to 1m to run my analysis. I generated a overlapping index grid allowing for 20% overlap and sliced the contour datasets into 500 tiles. I then generated a Python module utilising Python's Multiprocessing module to process the tiles across 16 cores on our servers. The DTM tiles were completed in under 12 hrs for the entire study area. The Python module which I've attached is below, is my first draft of utilizing Python's Multiprocessing. I'm currently re-writing it now that I have a better understanding how the Multiprocessing module works.

NB: I'd like to thank Duncan Hornby for his post: Create a script tool that uses multiprocessing, that helped me create the following script

Hope the following helps anyone else trying to achieve the same for generating large DTM's

Dar es Salaam: 1m DTM Tiles

'''
Created on 01 Feb 2018

Multi-threaded

Topo to Raster (3D)

@author: PeterW
'''
# import site-packages and modules
import multiprocessing
import argparse
import arcpy
from functools import partial

# set environment settings
arcpy.env.overwriteOutput = True

# check-out extensions
arcpy.CheckOutExtension('3D')


def dtm_work(tiles_directory, cont):
    """
    Multiprocess
    Topo to Rasster
    for multiple
    DTMs
    """
    index_name = cont[0]
    inContours = cont[1]
    try:
        output_dtm = '{0}\\{1}'.format(tiles_directory, index_name)
        arcpy.TopoToRaster_3d(inContours,
                     out_surface_raster=output_dtm,
                     cell_size=1)
        return True
    except Exception as e:
        print e.message
        return False


def dtm_multi(index_grid, tiles_directory):
    """
    Create overlapping
    DTM tiles from
    contour tiles &
    index grid
    """
    try:
        cont_param = []
        with arcpy.da.SearchCursor(index_grid, ['SHAPE@', 'PageName', 'Path']) as scur:  # @UndefinedVariable
            for row in scur:
                index_name = row[1]
                input_cont = '{0} {1} {2}'.format(row[2], 'ELEVATION', 'Contour')
                cont_param.append([index_name, input_cont])
        func = partial(dtm_work, tiles_directory)
        arcpy.AddMessage('Sending to pool')
        # declare the number of cores to use, use 2 less than the max
        cpu_num = multiprocessing.cpu_count()-2
        # create the pool object
        pool = multiprocessing.Pool(processes=cpu_num)
        # fire off list to worker function
        # res is a list that is created with what ever the worker function returns
        res = pool.map(func, cont_param)
        pool.close()
        pool.join()
        # if an error has occurred report it
        if False in res:
            arcpy.AddError('A process\thread failed!')
        arcpy.AddMessage('Finished Multiprocessing DTM Tiles')
    except arcpy.ExecuteError:
        # Geoprocessor through error
        arcpy.AddError(arcpy.GetMessages(2))


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Create overlapping DTM tiles for mosaic')
    parser.add_argument('--index_grid', metavar='path', required=True,
                        help='path to input overlapping index grid feature class')
    parser.add_argument('--tiles_directory', metavar='path', required=True,
                        help='path to output DTM tiles directory')
    args = parser.parse_args()
    dtm_multi(index_grid=args.index_grid,
              tiles_directory=args.tiles_directory)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Multi-threaded Topo to Raster

DanPatterson_Retired — Sun, 04 Feb 2018 19:01:38 GMT

Nice Peter... going to try this out.

topic Re: Multi-threaded Topo to Raster in Python Questions

Multi-threaded Topo to Raster

Re: Multi-threaded Topo to Raster