This is a duplicate post from the Hydro Forum: http://forums.arcgis.com/threads/55351-CPU-speed-and-Memory-requirements-for-ArcHydro-and-Geoprocessing I'm wondering what others are using to operate ArcHydro (performs geoprocessing for hydrology purposes). I'm getting more and more suspicious that most of the current problems I'm having stem from memory issues. I've got:* Windows 7, ArcGIS 10, the latest version of ArcHydro. * 64-bit * 4GB RAM * 300 GHz Duo Core Intel processor * I'm currently working with a 10' grid DEM 10.2 square miles mask extracted rawdem of 24.41 MB. Seems like plenty, but is this enough? This is the same configuration they have given everyone in the Department. I'd need "special dispensation" to get more.

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

CPU speed and Memory requirements for ArcHydro and Geoprocessing

MarkBoucher

This is a duplicate post from the Hydro Forum: http://forums.arcgis.com/threads/55351-CPU-speed-and-Memory-requirements-for-ArcHydro-and-Geoprocessing

I'm wondering what others are using to operate ArcHydro (performs geoprocessing for hydrology purposes).

I'm getting more and more suspicious that most of the current problems I'm having stem from memory issues.

I've got:

Windows 7, ArcGIS 10, the latest version of ArcHydro.
64-bit
4GB RAM
300 GHz Duo Core Intel processor
I'm currently working with a 10' grid DEM 10.2 square miles mask extracted rawdem of 24.41 MB.

Seems like plenty, but is this enough?

This is the same configuration they have given everyone in the Department. I'd need "special dispensation" to get more.

タグ付けされた投稿をもっと見る

Accepted answers

ChrisSnyder

Your system is entirely sufficient for using ArcGIS and ArcHydro... especially given that your dataset is relatively small. However, if you have some $$$ to blow:

1. Get the fastest/best processor you can afford. Right now (April, 2012), those would be the Intel Core i7 3960x (for a desktop), or an Intel Xeon E5-2687W (for a workstation/server).
2. Get a solid state hard drive... The Intel Cherryville 520 series is looking pretty good. Better yet, get 2 of these drives so you can use RAID0.

All comments

JohnGallo

Hi, Great Thread!

i made a blog post that includes it, some other info, and also pose a question. both linked from post 28 on http://forums.arcgis.com/threads/2873-Opinions-for-a-new-computer?p=225264&viewfull=1#post225264

MarkBoucher

Zakir,

My work around has been to NOT give up! I often (way to often) get this or other errors after attempting to create the fdr, fac, cat, str, strlnk, or cat layers (raster layers). What I do is I refresh the catalog. You will sometimes see three (3) files with the same name and different extensions. I keep refreshing the catalog view until I see the normal one file with the correct icon. I then look at the properties and see that the coordinate system is not set. I set it to what it should be, add it to the mxd and move on. This can be frustrating, and I don't know what's going on (why the error, but eventually the file is there, but it has no coordinate system) but it works.

Mark

Error:
[ATTACH=CONFIG]16160[/ATTACH]

After first refresh of catalog:
[ATTACH=CONFIG]16161[/ATTACH]

After second refresh of catalog:

[ATTACH=CONFIG]16162[/ATTACH]

After Second Refresh.jpg

After First Refresh.jpg

Error.jpg

zakirdahri

I followed your instructions yesterday (quoted above). Today I got back on a large rawdem project using ArcHydro. Still had the stream definition and stream link "fail".** I ran AdvancedArcMapSettings.exe and found that the setting was back to 65536 instead of 1000000 like I had set it. Does this need to be reset every time?

By the way, I've been running the same tools in Model Builder on a smaller watershed and its been working like a charm.

**(The model stops after the str and strlnk grids, but they are created without projections. When I add projections and put them in the MXD, then put them in the model and link them to the next steps, i.e. two manual tasks, the rest of model finished to adjoincatchments).

Hi,
I am also facing similar problem. My PC is with 4 GB RAM and I am using ArcGIS10 and ArcHydro2.0 and HEC-GeoHMS. both are not working. ArcHydro was able to calcluate Fill Sinks and Flow direction function but for flow accumulation it hanged. After closing the ArcGIS and deleting all temporary files, I rerun the flow accumulation, it gives the following message. (Pls see also attached file for detailed message).

System.Runtime.InteropServices.COMException (0x80004005): Error HRESULT E_FAIl has been returned from a call to a COM component.

I am using TIFF file and its size is 12700 col by 8500 rows. I need your help pls.

Zakir Dahri
PhD Student
Wageningen University, The Netherlands

ArcHydro_error_mes.docx

JeffTang

I'm not entirely sure how you came to the conclusion that there is some size limitation imposed by ArcGIS, but there is not. Certain formats have size limitations. Tiff comes to mind, but even the old 4gb limit has been overcome with the advent of BigTiff format which ArcGIS is capable of writing. I've been working on some hydro things since the beginning of the year, specifically processing NHDPlus v1 data for the lower 48 states. I routinely process data with ArcHydro that is in excess of 8-10GB. Outside of my hydro work, I do a lot of testing on NED data, which is several hundred GB's of data. 1GB of data is really nothing much these days.

My test machine specs:
Windows Server 2008 R2 64 bit, ArcGIS 10.1
6GB RAM
Intel Xeon W3550 @ 3.07GHz

Best Regards,
Eric

I was told by HEC, that there is a ArcGIS limit when processing rasters.

curtvprice

I tried to find the HKEY_LOCAL_MACHINE\SOFTWARE\ESRI\Raster\Preferences\grid.max_table_range in Registry Editor, but it was not there.

In Arc 10, this setting is part of the main interface.

Arc 10 Help: Setting default raster display options
http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//009t0000008z000000

EricRice

The errors were more likely due to size limitation of ArcGIS than my computer components. I've since found a workaround for the DEM where it's no longer a 1+ GB file (although my coworker recently informed me that it IS possible to process a DEM greater than 1GB). I would imagine your setup is quite adequate for what you're doing. For comparison purposes, the coworker that recently discovered it was possible to process a DEM greater than 1GB, is using a laptop that is more than several years old, which I would guess has a similar setup as your desktop.

I'm not entirely sure how you came to the conclusion that there is some size limitation imposed by ArcGIS, but there is not. Certain formats have size limitations. Tiff comes to mind, but even the old 4gb limit has been overcome with the advent of BigTiff format which ArcGIS is capable of writing. I've been working on some hydro things since the beginning of the year, specifically processing NHDPlus v1 data for the lower 48 states. I routinely process data with ArcHydro that is in excess of 8-10GB. Outside of my hydro work, I do a lot of testing on NED data, which is several hundred GB's of data. 1GB of data is really nothing much these days.

My test machine specs:

Windows Server 2008 R2 64 bit, ArcGIS 10.1
6GB RAM
Intel Xeon W3550 @ 3.07GHz

Best Regards,
Eric

JeffTang

Greetings Mark!

I recently switched over (several months ago) from an Intel Core 2 Duo 2.4GHz processor with 4GB to a Intel i7 3.4GHz with 8GB. Prior to the switch, I was running ArcGIS V9.3 under Windows 7-64; I'm presently running ArcGIS 10. I don't think I necessarily needed to switch, but at the time, I was running into errors trying to process a large DEM and thought it might be due to the amount of memory. The errors were more likely due to size limitation of ArcGIS than my computer components. I've since found a workaround for the DEM where it's no longer a 1+ GB file (although my coworker recently informed me that it IS possible to process a DEM greater than 1GB). I would imagine your setup is quite adequate for what you're doing. For comparison purposes, the coworker that recently discovered it was possible to process a DEM greater than 1GB, is using a laptop that is more than several years old, which I would guess has a similar setup as your desktop.

MarkBoucher

Don't edit the registry directly... Instead, use the "AdvancedArcMapSettings.exe" registry editor - typically stored in the C:\Program Files\ArcGIS\Desktop10.0\Utilities folder.

I followed your instructions yesterday (quoted above). Today I got back on a large rawdem project using ArcHydro. Still had the stream definition and stream link "fail".** I ran AdvancedArcMapSettings.exe and found that the setting was back to 65536 instead of 1000000 like I had set it. Does this need to be reset every time?

By the way, I've been running the same tools in Model Builder on a smaller watershed and its been working like a charm.

**(The model stops after the str and strlnk grids, but they are created without projections. When I add projections and put them in the MXD, then put them in the model and link them to the next steps, i.e. two manual tasks, the rest of model finished to adjoincatchments).

ChrisSnyder

Negative on the thumbdrive idea... The transfer speed is too slow and the drive capacity too small. Although some of their comoponents are similar, thumb drives are not the same thing as a SSD.

Check out:
http://www.amazon.com/Intel-Series-2-5-Inch-Solid-State-Drive/dp/B006VCP9G6/ref=sr_1_1?s=electronics&ie=UTF8&qid=1336666141&sr=1-1
or
http://www.amazon.com/Samsung-830-Series-MZ-7PC256N-Internal/dp/B005T3GPXY

Things change rapidly, but these are probably among the best SSDs you can get in May, 2012.

MarkBoucher

Would a large capacity thumb drive work for this?

MarkBoucher

Thanks so much!

ChrisSnyder

Don't edit the registry directly... Instead, use the "AdvancedArcMapSettings.exe" registry editor - typically stored in the C:\Program Files\ArcGIS\Desktop10.0\Utilities folder.

MarkBoucher

Chris,

Thanks for the reply. I'll look into getting a SSD.

I tried to find the HKEY_LOCAL_MACHINE\SOFTWARE\ESRI\Raster\Preferences\grid.max_table_range in Registry Editor, but it was not there. The document "Comprehensive Terrain Preprocessing Using Arc Hydro Tools" was written for 9.3. Is there a different registry setting for 10.0?

I'm using Windows 7, ArcGIS 10, the latest version of ArcHydro, 64-bit machine.

ChrisSnyder

An update: I have been doing some testing of a machine with some newer 6.0 Gb/s SSDs (solid state hard drives). WOW!!! Huge performance increase over the 10k RPM SAS drives I had been using. Depends on the operation, but typically the hydrologic tools and associated raster-based tools (such as zonal statistics) are running 2-3 times faster with the SSDs. Raster stuff tends to be very disk I/O intensive, so I guess this makes sense. Anyway - in addition to CPU and memory... think about upgrading to SSDs also.

MarkBoucher

Maybe I'm not telling anything new

Fernando

This is new to me.

Thanks for replying to the post and for reference material! I found it at http://support.esri.com/en/downloads/datamodel/detail/15.

Cheers!

Mark

FernandoVasconcellos

Hi Mark,

I have only a little more (6 GB RAM, intel i5 3.0GHz) and no problem at all with large datasets (You've helped me in the very first steps, weeks ago). I noticed others have said you should have no problems with your machine and your not so large dataset and I don't know if you have already got the answer you were looking for. But when I downloaded the ArcHydro in the ESRI ftp site you referred to me, I got some reference documents there. There's an excellent article from Djokic 2008 (he has more recently evaluated the ArcGIS 10 with ArcHydro 2.xyz; seems to be an ESRI's expert in ArcHydro). The article, "Comprehensive Terrain Preporcessing Using Arc Hydro Tools", the best you could get on the subjet, has a good introduction about hardware requirements and some tips on how to deal with memory limits. For example, "For most common computers that can run ArcGIS software reasonably well, DEMs bigger than 20,000 by 20,000 cells will cause problems and potentially produce uncertain results"; and also "HKEY_LOCAL_MACHINE\SOFTWARE\ESRI\Raster\Preferences\grid.max_table_range contains the maximum number of records that are allowed for creation of VAT (STA) for a grid. The default is 65.536. When processing large DEM, that number might be too small and the function will fail (especially when performing fill sinks operation). Increasing that number (e.g. to 1.000.000) might fix a problem." Maybe I'm not telling anything new, but I'm answering your thread just in case this article can be of any help to you and others!

Fernando Cabral
Posdoc fellow at Université du Québec en Outaouais

MarkBoucher

I'm very tempted to take a shot at Python for GIS.

BTW. I was just asked to determine the watershed on a very small area. Model Builder worked flawlessly. Seems to me that size matters.

ChrisSnyder

Does not use AGREE, but could easily be made to do so.

This example leverages the subprocessing module in Python to create pseudo parallel processes, which work to create a 'stream layer' from a base DEM. I wrote this script to work on data (a large LiDAR-based DEM) that has disparate "patches" (aka non-contiguous coverage). Also, WA State Plane South Feet is assumed.... And currently the parallel processing part is worthless to a contiguous DEM. The code as it currently exists would have to be altered to work on a raster with a single contiguous area of data (basically, just run the 'child' script)... Contiguous elevation data - in a hydrologic sort of sense - can't easily be "parallelized". Hopefully ESRI will fix their Basin tool so that it doesn't bomb on large datasets (#NIM033938: The Basin tool fails on large datasets). If they fix that, then these hydrologic functions would be easy to parallelize :cool:!

Hope this gives you some ideas...

lidar_hydro_child_v10.py.zip

lidar_hydro_master_v10.py.zip

MarkBoucher

Tip: Ditch model builder and learn Python scripting.

That was going through my mind even this AM. I've use Python for minor data transformations, but am not well versed in it; and I haven't used it for ArcGIS at all.

Has anyone already developed a Python Script for Basic Dendritic Terrain Processing? I guess I could try exporting the one the comes with the ArcHydro tools...

ChrisSnyder

Tip: Ditch model builder and learn Python scripting.

MarkBoucher

What issues are you having in particular? Just general slowness or crashes?

Issues I'm having have to do with certain tools not working consistently, especially when I try to use Model Builder.

I tend to iterate. I have an agree streams layer that comes from various sources. I have to process from rawdem to fill to find locations where agreestream and innerwalls layers need to be edited. After that I move on to completing the terrain processing and looking for other anomalies where I might need to modify these layers again; and then I iterate again. Then I have to start at the beginning again. Model Builder would help immensely if it would work all the time, but it crashes at stream segmentation and stream link and sometimes when creating the cat layer.

When processing the stream segmentation and stream link using the menus, these rasters are created, but not without getting an error. After each step they show up when I refresh the catalog, but they don't have a projection. I add a projection and drag them into the mxd and move on with little problems. This sometimes happens when creating the cat layer.

My first thought was that memory was being eaten up during the Model Builder runs since the processes are "continuous" in the Model run.

Recently, I've been doing different parts of the processing using several Models and that seems to be working "OK-ish". Still not satisfied though.

MarkBoucher

Thanks for the response. I'd love to get the latest and greatest, but as with most government orgs., getting "special attention" on my machine would be difficu

ChrisSnyder

MathewCoyle

Not sure about ArcHydro in particular, but even for heavy ArcGIS processing it should be enough to run. What issues are you having in particular? Just general slowness or crashes? 4GB of RAM is on the low side but as long as you aren't doing a lot of parallel processing it should be fine.

会話に参加する

サインインしたメンバーは投稿、更新のフォローなどができます。初めてですか？無料アカウントを登録してください。

クイックリンク

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.