Arc Hydro - Do you modify to Parallel Processing Settings?

[MarkBoucher]

Hello Arc Hydro users,

I'm curious if anyone regularly modifies the Parallel Processing Settings in the Analysis Environmental Settings. If so, has it made any difference in the speed and reduced crashes?

I have migrated to Arc Hydro in Arc Pro (v3.3.52636) after my department finally made the transition.

I have 16GB on my machine, though ESRI recommends 32GB available with 64 GB for “optimal” use. 

I have been struggling to get Arc Hydro processes that worked before in Arc Pro to work again. After clearing the cache and keeping the Temp file as empty as possible, and rebooting, some problems seem to have gone away (for now).

Mark

[DeanDjokic]

Mark,

Let us know how how your transition evolves.  In Arc Hydro, we do not do anything particular on parallel processing settings (we use whatever has been set at system/Pro level).  16GB of memory IS on the low end for any significant raster processing.  Depending on the tools you use (some load the full raster into memory and some can process it in "chunks") you might "struggle".  But in general, that should not be different between the Pro/AH versions (but could, if the core tools have been tweaked).  If you see consistent "misbehavior" with particular AH tool, let us know, and we'll check.

Dean

[MarkBoucher]

Dean,

Thanks for the reply; really appreciate your response.

I am generally conversant in the ArcMap version of Arc Hydro (AH) and the ArcPro version seems to be nearly the same. I typically only work with the AH tools when a project comes up. The latest project is a redo of one I cut my teeth on about 10 years ago. Unfortunately the layers from that effort wouldn't load into the latest version of ArcMap.

I did have similar struggles with the AH in ArcMap, but maybe that was this low RAM issue that I was unaware of before now.

So far I have run into a few things in the AH Pro version that seem different and require additional steps:

• The agreestream/breach line data now needs to be converted raster before DEM Reconditioning. This is easy to address with the polyline to raster tool.
• The Adjust Flow Direction in Streams tool now requires the Input Stream Flow Direction data raster as input. I'm struggling to make this work by converting the polylines with the correct direction into a d8 raster. I have been trying to do this in arcpy script, which I'm not perfectly conversant with. I have used AI tools (Gemini) help with the coding with some success, but didn't get it fine tuned last week. It was nice in the ArcMap version when I simply provided polyline data with the lines drawn in the right direction.

I am asking my IT group if they could upgrade my RAM. In our organization (in the past at least) tailoring computers has been very hard to achieve.

One idea I had was to add the "bookkeeping" effort of dividing the larger watershed into smaller working parts then merge the hydrologically correct DEMs together and do the rest from that. I do realize I could still struggle due to low RAM issues. I've attached a snapshot of my rawdem properties so you can see the basic size of my data. We have defined subwatersheds that I could use to break this into smaller parts.

FYI - I've always used ModelBuilder to put the processing steps together. It would be good to know if making models of the process increases the memory use or cause any other issues verses simply using the toolbox tools. There is no toolbar now as far as I can tell other than three tools under the Arc Hydro ribbon. Since there are pre-made models (such as Basic Dendritic Processing), I assume using ModelBuilder a reasonable practice.

Thanks,

Mark

[DeanDjokic · Accepted solution]

Mark,

As we move AH capabilities from ArcMap to Pro, we will tweak some of the tools and workflows (and drop tools and add some).  Within Pro world as well based on new Pro capabilities or enough experience that a better workflow is more stable/productive.  There are several points you are making in your notes.

1. Data processing for Arc Hydro that you have done 10 years ago should work in Pro (rasters, vectors, tables).  The basic data schema has not changed.  The only difference will be lack of support for geometric networks (replaced by trace network in Pro).  But even for that, the basic feature classes that you would use in the network would be the same and should work.  That does not mean that you can use the ArcMap .mxd from 10 years ago and just import it into Pro .aprx and that everything will work.  But if you take your gdb and raster layers derived using AH tools in ArcMap, add them into new Pro project, and use AH in Pro, they should “just work”.  That should also work for the ArcMap project – you just might have to rebuild the map, but the data should work (but I do suggest you use Pro 😊).
2. Correct on stream raster being the only type of input into agree.  Doing vector to raster conversion can sometimes generate inconsistent lines so it is safer to do it once and then use that raster layer consistently instead of letting the tools do it internally.  See next comment for an alternative and precise way of doing it that avoids inconsistencies.
3. The “Input Stream Flow Direction Raster” for the “Adjust Flow Direction in Streams” tool should use the results of the “Create Drainage Line Structure” tool.  This tool will take your stream line vector and produce both “clean” stream raster AND the stream flow direction raster.  So your typical workflow when you want to enforce known streams (vectors) into the DEM is to first run “Create Drainage Line Structure” and use the results from that tool as inputs in the rest of your terrain preprocessing workflow.
4. If you are new to using Pro, you should definitely look into the new core capabilities in Pro that produce flow direction and flow accumulation (and streams) without the need for filling sinks (“Derive Continuous Flow” in Hydrology toolset in SA – its help has a very detailed explanation of the method).  It uses least cost path to find its way out of depressions.  Not perfect, specially if the DEM was hydro-flattened, but in general produces better results than the traditional D8 approach.  We have that rolled into AH as well (albeit not that well documented in the workflow context yet).
5. “Divide and Conquer”.  Absolutely YES.  We have methodology that we call “Global Processing” that allows you to process “chunks” (watersheds of “appropriate” size) independently of each other and then assemble into a single processing system where you can do global watershed delineation and characterization.  That is behind Esri global watershed delineation and tracing services and USGS StreamStats.  Needs a bit of work up front, but you can then process any landscape size.
   1. BUT, based on the DEM characteristics you have sent (~ 8,000 by, 8,000 cells) that should NOT require global approach.  Something of that size should work even on a 16GB machine.  And not take more than 20 minutes to process.  So you might be really hamstrung buy your processing environment.  To (try to) work around that, try the following:
      1.       Do not use rasters in file geodatabase – use .tif format.
      2.       Close ALL applications except Pro (and I mean ALL).  Check your task manager and see if there are some lingering apps around including some background services that you might not be using.  Stop them all and see if that frees some memory/CPU cycles.
      3. Clear up the %temp% folder (you have been doing that already) and make sure you have at least 10 times the HD space of the DEM you are processing both on the drive you are storing the data on and where your %temp% folder is.
      4. Do NOT use network or shared drives (OK to use local external USB3 drives).
      5. Use individual tools instead of workflows.  It is tedious but running “superbuttons” (either MB or scripts) has overhead and when running on “fumes” everything counts.
      6. When all fails, see if you can find someone who has better equipment or if you can spin up a VM with better resources for the lifecycle of the project (I know, easier said than done ☹).
6. I mentioned above that any “superbutton” has some overhead.  Might not be a lot, but there is some.  Typically not a big issue, but again, when you are strapped for resources everything counts.  Another thing with automation is system memory management.  There is an inherent memory management at the OS level that decides when to do memory “garbage collection”.  In an automated process, that might happen later in the process, while if you are running one tool at the time, it will happen sooner (typically after the tool is complete and control returned to the system).  With all that said, I do recommend that you build your automation scripts.  Instead of using Model Builder, I suggest you do it in Python.  You have much better control over the process and have full Python universe at your disposal for inclusion in the script.  We are in the process of taking the model builder models in AH and replacing them with scripts.  Hopefully by the end of the year.  Model Builder is great for quick prototyping and “personal” automation, and can be a great steppingstone towards a final script. And so can be a Notebook.

Hope this helps.

Dean

[MarkBoucher]

Thanks for the detailed reply Dean. I'll take your words to heart and move forward.

Mark