Determine Flow Accumulation GRID: Using Python; Map Algebra and NumPy

[PeterWilson]

I'm currently busy with a hydrology study where I'm determining the location and structure requirements (culverts & bridges) for a proposed Railway Alignment. My initial assessment is based on the STRM 90m DEM and the size of my DEM is 34 651 X 24 934 cells = 863 988 034 cells. I can't afford to split my DEM or to increase the cells size as its already a coarse cell size. I've used the Flow Accumulation tool within Arc Hydro, the Flow Accumulation tool within the Spatial Analyst Extension; Hydrology Toolbox as well as tried running it from Python hoping for better processing time. The Flow Accumulation algorithm that is being used seems to be hard coded in that it doesn't use any more than 2 GB memory when processing the Flow Direction GRID to generate the Flow Accumulation GRID.

Is there a way to use Python; Map Algebra and Numpy to improve my processing time in generating the Flow Accumulation GRID for large DEM's.

The current Flow Accumulation GRID runs for almost 72 hrs.

Regards

[CarrieDavis]

Hi,

A bug was logged on this issue in version 10.0 as http://support.esri.com/en/bugs/nimbus/TklNMDYyNjg4.  I would suggest to contact support@esri.com to gain the full status of this bug report.

I would also like to make sure you are aware that as of version 10.1 SP1 you do have access to more memory when running processes in ArcMap and with python.

Python scripting with 64-bit processing
http://blogs.esri.com/esri/arcgis/2012/11/12/python-scripting-with-64-bit-processing/

[DuncanHornby]

Are you saying you rail realignment is over 3000Km long??? Your grid data is covering an area of 3118Km by 2244Km. It may be worth the effort to clip back your DEM to the area you are doing the analysis on.

[PeterWilson]

Dear Carrie\Duncan

I'm currently running ArcGIS 10.1 with 64bit Geoprocessing installed. The Flow Accumulation algorithm seems to do a lot of swappping between memory and disk while processing. It seems that the Flow Accumulation Algorithm has not been updated to processes directly from memory. I'm going to try to convert the Flow Direction GRID to a NumPy Array and see if I can improve the processing time.

Duncan, my study area is clipped to the smallest possible extent, the reason for the size is that the Abuja Basin flows through my three alignments and the basin is approximately 2.5 million km².

Regards

[PaulHedlund]

Any luck with using NumPy for Flow Accumulation?

[PeterWilson]

Hi Paul

I've not been able to improve the Arc Hydro Flow Accumulation processing, but I must give credit to the Arc Hydro Team. I'm using one of the latest versions of Arc Hydro and the processing time has drastically improved. I've processed a 500 000 000 million cells DEM study area within 2hrs.

I'm currently busy with my Masters Thesis and busy developing a Multi Flow Direction algorithm that will divide the study area into smaller tiles and process them simultaneously, improving the overall processing time and accuracy of the derive flow accumulation grid and derived river network.

Regards

[PaulHedlund]

Peter,

I compared processing times between the Arcpy Flow Accumulation and Arc Hydro Flow Accumulation.  They appear to be almost identical.

The multi-processing algorithm sounds interesting.