CPU speed and Memory requirements for ArcHydro and Geoprocessing

2224
24
Jump to solution
04-17-2012 06:41 AM
MarkBoucher
Regular Contributor II

This is a duplicate post from the Hydro Forum: http://forums.arcgis.com/threads/55351-CPU-speed-and-Memory-requirements-for-ArcHydro-and-Geoprocess...

I'm wondering what others are using to operate ArcHydro (performs geoprocessing for hydrology purposes).

I'm getting more and more suspicious that most of the current problems I'm having stem from memory issues.

I've got:



  • Windows 7, ArcGIS 10, the latest version of ArcHydro.

  • 64-bit

  • 4GB RAM

  • 300 GHz Duo Core Intel processor

  • I'm currently working with a 10' grid DEM 10.2 square miles mask extracted rawdem of 24.41 MB.

Seems like plenty, but is this enough?

This is the same configuration they have given everyone in the Department. I'd need "special dispensation" to get more.

0 Kudos
1 Solution

Accepted Solutions
ChrisSnyder
Regular Contributor III
Your system is entirely sufficient for using ArcGIS and ArcHydro... especially given that your dataset is relatively small. However, if you have some $$$ to blow:

1. Get the fastest/best processor you can afford. Right now (April, 2012), those would be the Intel Core i7 3960x (for a desktop), or an Intel Xeon E5-2687W (for a workstation/server).
2. Get a solid state hard drive... The Intel Cherryville 520 series is looking pretty good. Better yet, get 2 of these drives so you can use RAID0.

View solution in original post

0 Kudos
24 Replies
MathewCoyle
Frequent Contributor
Not sure about ArcHydro in particular, but even for heavy ArcGIS processing it should be enough to run. What issues are you having in particular? Just general slowness or crashes? 4GB of RAM is on the low side but as long as you aren't doing a lot of parallel processing it should be fine.
0 Kudos
ChrisSnyder
Regular Contributor III
Your system is entirely sufficient for using ArcGIS and ArcHydro... especially given that your dataset is relatively small. However, if you have some $$$ to blow:

1. Get the fastest/best processor you can afford. Right now (April, 2012), those would be the Intel Core i7 3960x (for a desktop), or an Intel Xeon E5-2687W (for a workstation/server).
2. Get a solid state hard drive... The Intel Cherryville 520 series is looking pretty good. Better yet, get 2 of these drives so you can use RAID0.
0 Kudos
MarkBoucher
Regular Contributor II
Thanks for the response. I'd love to get the latest and greatest, but as with most government orgs., getting "special attention" on my machine would be difficu
0 Kudos
MarkBoucher
Regular Contributor II
What issues are you having in particular? Just general slowness or crashes?


Issues I'm having have to do with certain tools not working consistently, especially when I try to use Model Builder.

I tend to iterate. I have an agree streams layer that comes from various sources. I have to process from rawdem to fill to find locations where agreestream and innerwalls layers need to be edited. After that I move on to completing the terrain processing and looking for other anomalies where I might need to modify these layers again; and then I iterate again. Then I have to start at the beginning again. Model Builder would help immensely if it would work all the time, but it crashes at stream segmentation and stream link and sometimes when creating the cat layer.

When processing the stream segmentation and stream link using the menus, these rasters are created, but not without getting an error. After each step they show up when I refresh the catalog, but they don't have a projection. I add a projection and drag them into the mxd and move on with little problems. This sometimes happens when creating the cat layer.

My first thought was that memory was being eaten up during the Model Builder runs since the processes are "continuous" in the Model run.

Recently, I've been doing different parts of the processing using several Models and that seems to be working "OK-ish". Still not satisfied though.
0 Kudos
ChrisSnyder
Regular Contributor III
Tip: Ditch model builder and learn Python scripting.
0 Kudos
MarkBoucher
Regular Contributor II
Tip: Ditch model builder and learn Python scripting.

That was going through my mind even this AM. I've use Python for minor data transformations, but am not well versed in it; and I haven't used it for ArcGIS at all.

Has anyone already developed a Python Script for Basic Dendritic Terrain Processing? I guess I could try exporting the one the comes with the ArcHydro tools...
0 Kudos
ChrisSnyder
Regular Contributor III
Does not use  AGREE, but could easily be made to do so.

This example leverages the subprocessing module in Python to create pseudo parallel processes, which work to create a 'stream layer' from a base DEM. I wrote this script to work on data (a large LiDAR-based DEM) that has disparate "patches" (aka non-contiguous coverage). Also, WA State Plane South Feet is assumed.... And currently the parallel processing part is worthless to a contiguous DEM. The code as it currently exists would have to be altered to work on a raster with a single contiguous area of data (basically, just run the 'child' script)... Contiguous elevation data - in a hydrologic sort of sense - can't easily be "parallelized". Hopefully ESRI will fix their Basin tool so that it doesn't bomb on large datasets (#NIM033938: The Basin tool fails on large datasets). If they fix that, then these hydrologic functions would be easy to parallelize :cool:!

Hope this gives you some ideas...
0 Kudos
MarkBoucher
Regular Contributor II
I'm very tempted to take a shot at Python for GIS.

BTW. I was just asked to determine the watershed on a very small area. Model Builder worked flawlessly. Seems to me that size matters.
0 Kudos
FernandoVasconcellos
New Contributor III
Hi Mark,

I have only a little more (6 GB RAM, intel i5 3.0GHz) and no problem at all with large datasets (You've helped me in the very first steps, weeks ago). I noticed others have said you should have no problems with your machine and your not so large dataset and I don't know if you have already got the answer you were looking for. But when I downloaded the ArcHydro in the ESRI ftp site you referred to me, I got some reference documents there. There's an excellent article from Djokic 2008 (he has more recently evaluated the ArcGIS 10 with ArcHydro 2.xxx; seems to be an ESRI's expert in ArcHydro). The article, "Comprehensive Terrain Preporcessing Using Arc Hydro Tools", the best you could get on the subjet, has a good introduction about hardware requirements and some tips on how to deal with memory limits. For example, "For most common computers that can run ArcGIS software reasonably well, DEMs bigger than 20,000 by 20,000 cells will cause problems and potentially produce uncertain results"; and also "HKEY_LOCAL_MACHINE\SOFTWARE\ESRI\Raster\Preferences\grid.max_table_range contains the maximum number of records that are allowed for creation of VAT (STA) for a grid. The default is 65.536. When processing large DEM, that number might be too small and the function will fail (especially when performing fill sinks operation). Increasing that number (e.g. to 1.000.000) might fix a problem." Maybe I'm not telling anything new, but I'm answering your thread just in case this article can be of any help to you and others!

Fernando Cabral
Posdoc fellow at Université du Québec en Outaouais
0 Kudos