Arc Hydro - Do you modify to Parallel Processing Settings?

MarkBoucher

 

Hello Arc Hydro users,

I'm curious if anyone regularly modifies the Parallel Processing Settings in the Analysis Environmental Settings. If so, has it made any difference in the speed and reduced crashes?

I have migrated to Arc Hydro in Arc Pro (v3.3.52636) after my department finally made the transition.

I have 16GB on my machine, though ESRI recommends 32GB available with 64 GB for “optimal” use. 

I have been struggling to get Arc Hydro processes that worked before in Arc Pro to work again. After clearing the cache and keeping the Temp file as empty as possible, and rebooting, some problems seem to have gone away (for now).

Mark

DeanDjokic

Mark,

As we move AH capabilities from ArcMap to Pro, we will tweak some of the tools and workflows (and drop tools and add some).  Within Pro world as well based on new Pro capabilities or enough experience that a better workflow is more stable/productive.  There are several points you are making in your notes.

1. Data processing for Arc Hydro that you have done 10 years ago should work in Pro (rasters, vectors, tables).  The basic data schema has not changed.  The only difference will be lack of support for geometric networks (replaced by trace network in Pro).  But even for that, the basic feature classes that you would use in the network would be the same and should work.  That does not mean that you can use the ArcMap .mxd from 10 years ago and just import it into Pro .aprx and that everything will work.  But if you take your gdb and raster layers derived using AH tools in ArcMap, add them into new Pro project, and use AH in Pro, they should “just work”.  That should also work for the ArcMap project – you just might have to rebuild the map, but the data should work (but I do suggest you use Pro 😊).
2. Correct on stream raster being the only type of input into agree.  Doing vector to raster conversion can sometimes generate inconsistent lines so it is safer to do it once and then use that raster layer consistently instead of letting the tools do it internally.  See next comment for an alternative and precise way of doing it that avoids inconsistencies.
3. The “Input Stream Flow Direction Raster” for the “Adjust Flow Direction in Streams” tool should use the results of the “Create Drainage Line Structure” tool.  This tool will take your stream line vector and produce both “clean” stream raster AND the stream flow direction raster.  So your typical workflow when you want to enforce known streams (vectors) into the DEM is to first run “Create Drainage Line Structure” and use the results from that tool as inputs in the rest of your terrain preprocessing workflow.
4. If you are new to using Pro, you should definitely look into the new core capabilities in Pro that produce flow direction and flow accumulation (and streams) without the need for filling sinks (“Derive Continuous Flow” in Hydrology toolset in SA – its help has a very detailed explanation of the method).  It uses least cost path to find its way out of depressions.  Not perfect, specially if the DEM was hydro-flattened, but in general produces better results than the traditional D8 approach.  We have that rolled into AH as well (albeit not that well documented in the workflow context yet).
5. “Divide and Conquer”.  Absolutely YES.  We have methodology that we call “Global Processing” that allows you to process “chunks” (watersheds of “appropriate” size) independently of each other and then assemble into a single processing system where you can do global watershed delineation and characterization.  That is behind Esri global watershed delineation and tracing services and USGS StreamStats.  Needs a bit of work up front, but you can then process any landscape size.
   1. BUT, based on the DEM characteristics you have sent (~ 8,000 by, 8,000 cells) that should NOT require global approach.  Something of that size should work even on a 16GB machine.  And not take more than 20 minutes to process.  So you might be really hamstrung buy your processing environment.  To (try to) work around that, try the following:
      1.       Do not use rasters in file geodatabase – use .tif format.
      2.       Close ALL applications except Pro (and I mean ALL).  Check your task manager and see if there are some lingering apps around including some background services that you might not be using.  Stop them all and see if that frees some memory/CPU cycles.
      3. Clear up the %temp% folder (you have been doing that already) and make sure you have at least 10 times the HD space of the DEM you are processing both on the drive you are storing the data on and where your %temp% folder is.
      4. Do NOT use network or shared drives (OK to use local external USB3 drives).
      5. Use individual tools instead of workflows.  It is tedious but running “superbuttons” (either MB or scripts) has overhead and when running on “fumes” everything counts.
      6. When all fails, see if you can find someone who has better equipment or if you can spin up a VM with better resources for the lifecycle of the project (I know, easier said than done ☹).
6. I mentioned above that any “superbutton” has some overhead.  Might not be a lot, but there is some.  Typically not a big issue, but again, when you are strapped for resources everything counts.  Another thing with automation is system memory management.  There is an inherent memory management at the OS level that decides when to do memory “garbage collection”.  In an automated process, that might happen later in the process, while if you are running one tool at the time, it will happen sooner (typically after the tool is complete and control returned to the system).  With all that said, I do recommend that you build your automation scripts.  Instead of using Model Builder, I suggest you do it in Python.  You have much better control over the process and have full Python universe at your disposal for inclusion in the script.  We are in the process of taking the model builder models in AH and replacing them with scripts.  Hopefully by the end of the year.  Model Builder is great for quick prototyping and “personal” automation, and can be a great steppingstone towards a final script. And so can be a Notebook.

Hope this helps.

Dean