topic Re: DEM Reconditioning >> Empty AgreeDEM in Water Resources Questions

DEM Reconditioning >> Empty AgreeDEM

RyanLondeen — Thu, 26 Feb 2015 15:20:41 GMT

I am having issue with the terrain processing with ArcHydro.

ArcGIS Version: 10.2.2

ArcHydro Version: 10.2

Issue: Whenever I run the DEM Reconditioning tool I get an empty Agree DEM as an output.

My Steps:

1. Create project in a folder located on my Desktop (no spaces and relatively short file path)

2. Import master DEM raster for County

3. Import stream network for county (FEMA)

4. Create a polygon to setup mask boundary for project location.

5. Extract by mask the DEM to boundary

6. Clip wtr to boundary.

7. Create file geodatabase

8. Set extents to boundary layer

9. Set Target locations to job folder and geodatabase

10. Execute DEM Reconditioning

I am I doing something wrong in the process?

Re: DEM Reconditioning >> Empty AgreeDEM

MarkBoucher — Thu, 26 Feb 2015 15:44:09 GMT

Ryan,

Whenever I've had empty results, the coordinate system seems to have been the culprit. Make sure you import the master DEM and then saving the mxd. This sets the coordinate system for the data frame. If you save it before you load data, the data frame will take on the coordinate system of the first layer added and you need to save the mxd again. It is important that all layers be on the same coordinate system. I'm not sure if when you add a layer that is in a different projection if the "on-the-fly" reprojection will avoid this problem.

Step 6: Is "wtr" your stream layer? I use "agreestream" as the layer name since this is the ArcHydro default stream layer used for burning the streams during the DEM Reconditioning. Leaving it as the default saves me time in having to set names in the processing steps. Also, I found putting the agreestream layer in a separate geodatabase adds a level of "stability". I have one for the entire county that I don't clip for an individual watershed processing. The size does not seem to matter much. I think the processing extents are limited to the DEM

Step 7 thru 9 : Arc Hydro will set the Target Locations and create the geodatabases and folders for you using the mxd and the data frame names. The only time I find I need to reset these is if I start the Arc Hydro processing and then change the name of the file or the data frame or if I use Save As to save a version of your work.

Hope this makes sense and helps.

Mark

Re: DEM Reconditioning >> Empty AgreeDEM

RyanLondeen — Thu, 26 Feb 2015 16:14:08 GMT

Mark:

Thank you for the tips. However, after making those changes, I still get an empty AgreeDEM.

- created project.

- imported DEM and saved.

- "agreestream" is in one geodatabase.

- "GT_LiDAR" (extracted mask) is in its own geodatabase.

- extents set to "Boundary" layer

- AgreeDEM - C:\Users\rlondeen\Desktop\GT\Layers\

I am starting to think it may be an issue with my data, but I am not sure how to identify if that is the issue.

I have attached an image of what I am getting.

Re: DEM Reconditioning >> Empty AgreeDEM

MarkBoucher — Thu, 26 Feb 2015 16:41:45 GMT

Hmmm... The rawdem that I use is in GRID format with floating point. I converted it to tiff, jpg, and bmp (most common formats) using the Raster To Other Format (multiple) tool then ran the DEM Reconditioning on all of them. All of them worked without a problem. I you have your DEM in some other format, you could try to convert it to GRID and see if that works.

The next thing I would do, if you didn't do this already, is start from a brand new mxd. If you've done that and it still fails, before you do the DEM Reconditioning, go to Geoprocessing>Environments>Processing Extents and choose your "Boundary" layer to set the processing extents. Then run it. If this works, there is some glitch somewhere and the processing extent is not being set automatically. I can't imagine that ESRI wouldn't have this set either in the Arc Hydro coding or some other place in the overall scheme of things. I have had the extent reduced to the extents of the BatchPoint data and have had to reset it at that point (probably due do crashing partway through some process).

After this I'd need to get a copy of your data and try it myself. I am using 10.1, not 10.2, so my doing a test may not help.

Mark

Re: DEM Reconditioning >> Empty AgreeDEM

RyanLondeen — Thu, 26 Feb 2015 18:07:19 GMT

Mark:

- My original DEM was in a file geodatabase and had is called a File Geodatabase Raster Dataset (FGDBR). I took that DEM, exported it, and turned it into a GRID. Again, empty AgreeDEM.

- I also tried creating a raster from a point file that i had (points > tin > raster). Again, an empty AgreeDEM.

- I tried changing my geoprocessing extents to "default" and it gave me an error.

If you don't have any other suggestions, then I can try to send you my files. This is not the first time I have had this issue. Its a reoccurring annoyance and I cannot remember why or how it worked in the past.

Re: DEM Reconditioning >> Empty AgreeDEM

RyanLondeen — Thu, 26 Feb 2015 18:48:24 GMT

Mark:

I tried it on a coworkers computer and it worked just fine. We are thinking it is a memory issue.

Re: DEM Reconditioning >> Empty AgreeDEM

GaneshchandraMallya — Thu, 11 Jun 2015 23:07:38 GMT

Hello Mark:

I ran into an issue almost similar to the one described by Ryan. In my case the AgreeDEM outputs without any error messages, but as seen from the attached image, more than half of the AgreeDEM has No-Data. Interestingly the pixels representing the stream is present in the No-Data zone.

I have followed your tips described as a probable solution to Ryan's question, and I have also followed some of the tips you have shared in your popular ArcHydro thread.

I am using ArcMap 10.2.2 on a Win 64 bit machine.

My data frame has a UTM-15N projected coordinate system and the target locations have been properly assigned using ApUtilities. The processing extent has also been checked.

My study area is the Upper Mississippi basin.

Kindly let me know if you have any thoughts on this issue.

Regards,

Ganesh

PS: I was previously successful in getting the AgreeDEM for the same area, on the same machine and ArcMap version. I have to repeat the process again as I have a higher resolution stream network to burn. Same issue encountered when I use my old stream network file.

Re: DEM Reconditioning >> Empty AgreeDEM

MarkBoucher — Fri, 12 Jun 2015 00:08:15 GMT

Ganesh,

Did you reset your processing extents via Geoprocessing>Environments>Processing Extents as I mention in my Feb 26, 2015 8:52 AM post to Ryan (above). When I see a raster "cut off" like that, my first thought is the processing extents got messed up. My most memorable situation was when the limits of the resulting dem were a rectangle that matched the extents of the BatchPoints I was using. I'm not sure what part of the Arc Hydro process might change the processing extents.

However, it could be that the values are NOT cut off. The white area could be positive numbers in your highest elevation classification. If you classify with a different color ramp you will likely see that the white area is not "blank" as happens when the extents are wrong. Maybe one of the intermittent steps resulted in a cut of one of the intermittent rasters due to the extents changing and subsequent steps reflect that "missing" part of that cut off DEM.

In your case something looks really strange because your elevation values in the AgreeDEM are very negative in general and you have "spots" all over that are dark making me think they are negative. I've never seen this before.

A step I take sometimes is to use the Spatial Analyst tools like raster calculator on the SA toolbar (you have to add it to the toolbar) to subtract one raster from the other (Agree DEM - rawdem). In a normal situation this will show you where the process has raised or filled in the DEM. This is really handy to see places you may need to add a stream such as at a culvert embankment that "blocks" a stream. Occasionally this will reveal something you couldn't see otherwise.

Sorry I can't be of more specific help. If I were you I'd make sure to save the agreestream, innerwalls, outerwalls, batchpoints and rawdem layers, etc., and delete everything else and start in a fresh mxd. When I do this I delete the folders, geodatabases and all. This would make sure the extents, tempfile locations, target locations, etc. are all set. They should be setup again automatically by the Arc Hydro process. Coordinate systems need to also match. I have not dealt with a vertical datum conflict issue and that is one very remote possibility, especially seeing how your elevations "inverted" for some reason.

Best,

Mark

Re: DEM Reconditioning >> Empty AgreeDEM

MarkBoucher — Fri, 12 Jun 2015 00:17:58 GMT

Ryan,

Can you upload the raster file or attach it to a post? I see an option under my profile icon to upload a file to share. I've not done this before, but it may be the best first option.

Mark

Re: DEM Reconditioning >> Empty AgreeDEM

GaneshchandraMallya — Fri, 12 Jun 2015 15:16:32 GMT

Hi Mark:

I was able to successfully obtain the AgreeDEM when I tried it again today.

Here is what I did (mostly following your tips in this thread as well as your popular ArcHydro thread):

Use Windows 7 utility Disk Cleanup to empty the Temp folder.
Open a new ArcMap document and add the DEM raster and the stream network shapefile.
Ensure that both these files (from step 2) have the same projection. In my case it was UTM 15N.
Create a folder for the project e.g. C:\MyProject
Save the ArcMap document as C:\MyProject\MyProject.mxd. This step generally sets the target locations correctly.
Double-check the target locations before running any ArcHydro functions using ArcHydro toolbar -> ApUtilities -> Set Target Locations -> HydroConfig. When you double click the HydroConfig
- Raster Data should be pointing to C:\MyProject\
- Vector Data should be pointing to C:\MyProject\MyProject.gdb
- Map Name = Layers (Layers is the default name of the dataframe in the ArcMap document)
Check the processing extent using Geoprocessing -> Environments -> Processing Extent
- I set the extent as Union of Inputs
Clear Temp folder again using ArcHydro toolbar -> ApUtilities -> Addition Utilities -> Clean User's Temp Folder
Run DEM Reconditioning to obtain AgreeDEM.

For the sake of any future GIS enthusiasts who encounter a similar error, I thought I will complete the discussion:

In my opinion, the original issue was not likely due to incorrect processing extents because the AgreeDEM (see attached figure from my initial post) did burn the stream network for the entire study area (burnt streams represented as black pixels spread over the study area- figure is at maximum zoom-out level). However, the entire left half (you can observe a vertical demarcation in the AgreeDEM - showing white region with black pixels on the LHS and a shaded region on the RHS of the AgreeDEM) of the study area (except the pixels burnt as stream) had pixels with value=No_Data. You rightly recommended me to check this again as the legend in the figure was showing white areas to be of high elevations (272-592). Though not shown in the legend, I have confirmed that most of the left half of the AgreeDEM had No_Data.

The values in the AgreeDEM appear to be strange at first look because by default the Symbology (or legend) for AgreeDEM was set to be Classified with inverted color coding compared to the input DEM. Further the AgreeDEM was run with the following parameters: Smooth drop in Z units = 100, and Sharp drop in Z units = 1000 to exaggerate the burning of pixels representing the stream. Same parameters were eventually used in the solution.

I am not able to pinpoint the exact cause for this error, except to agree that one needs to be careful while setting up the project and follow the best practices recommended by experts like you. Finally, if everything fails remember not to give up. Seek help. Start fresh.

I want to thank you again for your insights and detailed explanations for probable causes for this error. It was very helpful.

Best.

Ganesh

Re: DEM Reconditioning >> Empty AgreeDEM

MarkBoucher — Fri, 12 Jun 2015 20:11:51 GMT

Ganesh,

I'm very happy you were able to get it to work. Nice to know that others are contributors regarding Arc Hydro.

I'm no expert unless you consider an expert someone who is willing to wrestle with the pig until he figures out all the tricks to get it under control.

Happy pig wrestling,

Mark