topic Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing in Geoprocessing Questions

CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Tue, 17 Apr 2012 13:41:31 GMT

This is a duplicate post from the Hydro Forum: http://forums.arcgis.com/threads/55351-CPU-speed-and-Memory-requirements-for-ArcHydro-and-Geoprocessing

I'm wondering what others are using to operate ArcHydro (performs geoprocessing for hydrology purposes).

I'm getting more and more suspicious that most of the current problems I'm having stem from memory issues.

I've got:

Windows 7, ArcGIS 10, the latest version of ArcHydro.

64-bit

4GB RAM

300 GHz Duo Core Intel processor

I'm currently working with a 10' grid DEM 10.2 square miles mask extracted rawdem of 24.41 MB.

Seems like plenty, but is this enough?

This is the same configuration they have given everyone in the Department. I'd need "special dispensation" to get more.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MathewCoyle — Tue, 17 Apr 2012 17:46:18 GMT

Not sure about ArcHydro in particular, but even for heavy ArcGIS processing it should be enough to run. What issues are you having in particular? Just general slowness or crashes? 4GB of RAM is on the low side but as long as you aren't doing a lot of parallel processing it should be fine.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

ChrisSnyder — Thu, 26 Apr 2012 14:12:07 GMT

Your system is entirely sufficient for using ArcGIS and ArcHydro... especially given that your dataset is relatively small. However, if you have some $$$ to blow:

1. Get the fastest/best processor you can afford. Right now (April, 2012), those would be the Intel Core i7 3960x (for a desktop), or an Intel Xeon E5-2687W (for a workstation/server).
2. Get a solid state hard drive... The Intel Cherryville 520 series is looking pretty good. Better yet, get 2 of these drives so you can use RAID0.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Thu, 26 Apr 2012 14:15:40 GMT

Thanks for the response. I'd love to get the latest and greatest, but as with most government orgs., getting "special attention" on my machine would be difficu

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Thu, 26 Apr 2012 14:31:12 GMT

What issues are you having in particular? Just general slowness or crashes?

Issues I'm having have to do with certain tools not working consistently, especially when I try to use Model Builder.

I tend to iterate. I have an agree streams layer that comes from various sources. I have to process from rawdem to fill to find locations where agreestream and innerwalls layers need to be edited. After that I move on to completing the terrain processing and looking for other anomalies where I might need to modify these layers again; and then I iterate again. Then I have to start at the beginning again. Model Builder would help immensely if it would work all the time, but it crashes at stream segmentation and stream link and sometimes when creating the cat layer.

When processing the stream segmentation and stream link using the menus, these rasters are created, but not without getting an error. After each step they show up when I refresh the catalog, but they don't have a projection. I add a projection and drag them into the mxd and move on with little problems. This sometimes happens when creating the cat layer.

My first thought was that memory was being eaten up during the Model Builder runs since the processes are "continuous" in the Model run.

Recently, I've been doing different parts of the processing using several Models and that seems to be working "OK-ish". Still not satisfied though.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

ChrisSnyder — Thu, 26 Apr 2012 15:56:40 GMT

Tip: Ditch model builder and learn Python scripting.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Thu, 26 Apr 2012 17:43:37 GMT

Tip: Ditch model builder and learn Python scripting.

That was going through my mind even this AM. I've use Python for minor data transformations, but am not well versed in it; and I haven't used it for ArcGIS at all.

Has anyone already developed a Python Script for Basic Dendritic Terrain Processing? I guess I could try exporting the one the comes with the ArcHydro tools...

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

ChrisSnyder — Thu, 26 Apr 2012 18:00:38 GMT

Does not use AGREE, but could easily be made to do so.

This example leverages the subprocessing module in Python to create pseudo parallel processes, which work to create a 'stream layer' from a base DEM. I wrote this script to work on data (a large LiDAR-based DEM) that has disparate "patches" (aka non-contiguous coverage). Also, WA State Plane South Feet is assumed.... And currently the parallel processing part is worthless to a contiguous DEM. The code as it currently exists would have to be altered to work on a raster with a single contiguous area of data (basically, just run the 'child' script)... Contiguous elevation data - in a hydrologic sort of sense - can't easily be "parallelized". Hopefully ESRI will fix their Basin tool so that it doesn't bomb on large datasets (#NIM033938: The Basin tool fails on large datasets). If they fix that, then these hydrologic functions would be easy to parallelize :cool:!

Hope this gives you some ideas...

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Thu, 26 Apr 2012 22:07:08 GMT

I'm very tempted to take a shot at Python for GIS.

BTW. I was just asked to determine the watershed on a very small area. Model Builder worked flawlessly. Seems to me that size matters.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

FernandoVasconcellos — Wed, 09 May 2012 17:00:02 GMT

Hi Mark,

I have only a little more (6 GB RAM, intel i5 3.0GHz) and no problem at all with large datasets (You've helped me in the very first steps, weeks ago). I noticed others have said you should have no problems with your machine and your not so large dataset and I don't know if you have already got the answer you were looking for. But when I downloaded the ArcHydro in the ESRI ftp site you referred to me, I got some reference documents there. There's an excellent article from Djokic 2008 (he has more recently evaluated the ArcGIS 10 with ArcHydro 2.xxx; seems to be an ESRI's expert in ArcHydro). The article, "Comprehensive Terrain Preporcessing Using Arc Hydro Tools", the best you could get on the subjet, has a good introduction about hardware requirements and some tips on how to deal with memory limits. For example, "For most common computers that can run ArcGIS software reasonably well, DEMs bigger than 20,000 by 20,000 cells will cause problems and potentially produce uncertain results"; and also "HKEY_LOCAL_MACHINE\SOFTWARE\ESRI\Raster\Preferences\grid.max_table_range contains the maximum number of records that are allowed for creation of VAT (STA) for a grid. The default is 65.536. When processing large DEM, that number might be too small and the function will fail (especially when performing fill sinks operation). Increasing that number (e.g. to 1.000.000) might fix a problem." Maybe I'm not telling anything new, but I'm answering your thread just in case this article can be of any help to you and others!

Fernando Cabral
Posdoc fellow at Université du Québec en Outaouais

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Wed, 09 May 2012 17:14:30 GMT

Maybe I'm not telling anything new

Fernando

This is new to me.

Thanks for replying to the post and for reference material! I found it at http://support.esri.com/en/downloads/datamodel/detail/15.

Cheers!

Mark

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

ChrisSnyder — Wed, 09 May 2012 17:22:13 GMT

An update: I have been doing some testing of a machine with some newer 6.0 Gb/s SSDs (solid state hard drives). WOW!!! Huge performance increase over the 10k RPM SAS drives I had been using. Depends on the operation, but typically the hydrologic tools and associated raster-based tools (such as zonal statistics) are running 2-3 times faster with the SSDs. Raster stuff tends to be very disk I/O intensive, so I guess this makes sense. Anyway - in addition to CPU and memory... think about upgrading to SSDs also.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Wed, 09 May 2012 17:49:02 GMT

Chris,

Thanks for the reply. I'll look into getting a SSD.

I tried to find the HKEY_LOCAL_MACHINE\SOFTWARE\ESRI\Raster\Preferences\grid.max_table_range in Registry Editor, but it was not there. The document "Comprehensive Terrain Preprocessing Using Arc Hydro Tools" was written for 9.3. Is there a different registry setting for 10.0?

I'm using Windows 7, ArcGIS 10, the latest version of ArcHydro, 64-bit machine.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

ChrisSnyder — Wed, 09 May 2012 17:50:51 GMT

Don't edit the registry directly... Instead, use the "AdvancedArcMapSettings.exe" registry editor - typically stored in the C:\Program Files\ArcGIS\Desktop10.0\Utilities folder.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Wed, 09 May 2012 18:02:57 GMT

Thanks so much!

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Thu, 10 May 2012 14:25:49 GMT

Would a large capacity thumb drive work for this?

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

ChrisSnyder — Thu, 10 May 2012 15:10:07 GMT

Negative on the thumbdrive idea... The transfer speed is too slow and the drive capacity too small. Although some of their comoponents are similar, thumb drives are not the same thing as a SSD.

Check out:
http://www.amazon.com/Intel-Series-2-5-Inch-Solid-State-Drive/dp/B006VCP9G6/ref=sr_1_1?s=electronics&ie=UTF8&qid=1336666141&sr=1-1
or
http://www.amazon.com/Samsung-830-Series-MZ-7PC256N-Internal/dp/B005T3GPXY

Things change rapidly, but these are probably among the best SSDs you can get in May, 2012.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher — Thu, 10 May 2012 17:01:13 GMT

Don't edit the registry directly... Instead, use the "AdvancedArcMapSettings.exe" registry editor - typically stored in the C:\Program Files\ArcGIS\Desktop10.0\Utilities folder.

I followed your instructions yesterday (quoted above). Today I got back on a large rawdem project using ArcHydro. Still had the stream definition and stream link "fail".** I ran AdvancedArcMapSettings.exe and found that the setting was back to 65536 instead of 1000000 like I had set it. Does this need to be reset every time?

By the way, I've been running the same tools in Model Builder on a smaller watershed and its been working like a charm.

**(The model stops after the str and strlnk grids, but they are created without projections. When I add projections and put them in the MXD, then put them in the model and link them to the next steps, i.e. two manual tasks, the rest of model finished to adjoincatchments).

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

JeffTang — Thu, 14 Jun 2012 17:30:05 GMT

Greetings Mark!

I recently switched over (several months ago) from an Intel Core 2 Duo 2.4GHz processor with 4GB to a Intel i7 3.4GHz with 8GB. Prior to the switch, I was running ArcGIS V9.3 under Windows 7-64; I'm presently running ArcGIS 10. I don't think I necessarily needed to switch, but at the time, I was running into errors trying to process a large DEM and thought it might be due to the amount of memory. The errors were more likely due to size limitation of ArcGIS than my computer components. I've since found a workaround for the DEM where it's no longer a 1+ GB file (although my coworker recently informed me that it IS possible to process a DEM greater than 1GB). I would imagine your setup is quite adequate for what you're doing. For comparison purposes, the coworker that recently discovered it was possible to process a DEM greater than 1GB, is using a laptop that is more than several years old, which I would guess has a similar setup as your desktop.

Re: CPU speed and Memory requirements for ArcHydro and Geoprocessing

EricRice — Thu, 14 Jun 2012 20:39:53 GMT

The errors were more likely due to size limitation of ArcGIS than my computer components. I've since found a workaround for the DEM where it's no longer a 1+ GB file (although my coworker recently informed me that it IS possible to process a DEM greater than 1GB). I would imagine your setup is quite adequate for what you're doing. For comparison purposes, the coworker that recently discovered it was possible to process a DEM greater than 1GB, is using a laptop that is more than several years old, which I would guess has a similar setup as your desktop.

I'm not entirely sure how you came to the conclusion that there is some size limitation imposed by ArcGIS, but there is not. Certain formats have size limitations. Tiff comes to mind, but even the old 4gb limit has been overcome with the advent of BigTiff format which ArcGIS is capable of writing. I've been working on some hydro things since the beginning of the year, specifically processing NHDPlus v1 data for the lower 48 states. I routinely process data with ArcHydro that is in excess of 8-10GB. Outside of my hydro work, I do a lot of testing on NED data, which is several hundred GB's of data. 1GB of data is really nothing much these days.

My test machine specs:

Windows Server 2008 R2 64 bit, ArcGIS 10.1

6GB RAM

Intel Xeon W3550 @ 3.07GHz

Best Regards,
Eric