3D Analyst : TerrainToRaster, Good job!

2893
9
12-02-2009 10:40 PM
WentaoChe
Occasional Contributor II
I created a terrain from a Lidar point clouds with 1.65 billion points, then convert this terrain to raster, but failed in ArcGIS9.3.1 with an error of Out of Memory. The size of terrain is about 12GB.

I then try it in 9.4, I got what I want, that is Great! So far, I have to separate my Lidar dataset to several parts since TerrainToRaster tool fails.

Could someone tell me the limits of this tool?

Wentao
KKC, Tokyo, Japan
Tags (1)
9 Replies
ClaytonCrawford
Esri Contributor
Wentao,

There are no fixed limits for this tool. There are practical limits in terms of terrain size and output raster size. How much disk space are you willing to dedicate to one dataset? How long are you willing an individual process to run on the terrain or on the big raster you make from it?

If you're using 9.3.1 for production work - we have introduced a GP environment variable in 9.3.1 SP1 (shipping soon) that can be used as a workaround if you get an out of memory error. You'll find it under the 'Terrain Dataset' category on the GP Environments dialog and is listed as 'Minimize memory use during analysis on terrains'. In script the variable is 'terrainMemoryUsage' and is FALSE by default. You set to TRUE to turn the feature on. 9.4 solves the problem without relying on this.

By the way, we also confirmed another report of yours about the Natural Neighbors tool writing a scratch TIN to disk unnecessarily. It's fixed in 9.3.1 SP1 (and 9.4).

Regards, Clayton
0 Kudos
WentaoChe
Occasional Contributor II
Clayton,

Thank you so much for your reply. It is nice to know that there are no fixed limits for TerrainToRaster tool.

I like to run the terrain process for one project which usually covers one river basin or one administration boundary, the largest one we have done so far is about 2500 square kilometers, I process the data in a 1000 x 1000 meter tile with a 100 meter buffer since our clients ask for 1 meter DEM in tiles, I do not need to create one big raster for this case. When our clients asks for contour, one big raster is needed since each contour line must be connected physically, so tile process is not possible for this case.

Our 2m DEM is asked to cover one Satellite scene which is about 1000 square kilometers this time, it took me 12 hours to create Terrain, and this is just good for me because you can run it overnight and got it in the morning, and then you spend 7 hours to convert Terrain to Raster, this is also endurable, you got it before you go home, but it is better if you can make it finish within 4 hours ( half a working day). Please tell me the way to do so if you know.

I will try your workaround after I got 9.3.1 SP1 to reset my GP environment variable in my Python script.

It is nice to know that Natural Neighbors tool will not write a scratch TIN to disk unnecessarily in 9.3.1 SP1 and 9.4.

As I know Natural Neighbors tool does not work if your data is over about 8 millions points and give you an error of Out of Memory. Does ESRI intend to improve this tool, I hope that this tool can process 20 million points one time since this is the limit for one tile by the fixed wing aircraft, this could be 40 million for helicopter, of course I can use terrain instead.

Regards,

Wentao
KKC, Tokyo, Japan
0 Kudos
ClaytonCrawford
Esri Contributor
Wentao,

There is no way to speed up TerrainToRaster while keeping extent, cellsize, interpolation method, and terrain pyramid resolution the same. The 9.3.1sp1 workaround for the out of memory problem may actually slow things down a bit but there's no way around that.

You can speed up the terrain build process significantly using a windowsize pyramid filter rather than z-tolerance. This isn't going to be detrimental when using TerrainToRaster if you're rasterizing the full resolution data (i.e., pyramid resolution=0).

The Natural Neighbors tool is limited by TIN. The answer to support larger datasets is terrain. I realize the terrain build process adds time but if you use the windowsize pyramid filter, and define just one pyramid level that has a large window, the build process is pretty fast.

Regards, Clayton
0 Kudos
WentaoChe
Occasional Contributor II
Clayton,

I tried 2 meter Windowsize pyramid filter to build the terrain. It did speed up the terrain build process significantly, it took me only 3 hours compared to the z-tolerance with 11 hours. The terrain created by Windowsize using 1.79 billions points, much more points are used than Z-tolerance (1.65 billions), which one is better to present the true earth surface?

Do you think that I can use 2 meter Windowsize pyramid filter to create a 1m-DEM (bare earth), what Method should I use, ZMinimum or ZMean?

I will test the rasters created by both methods and report.

I found that the Tile Size of all my terrains I created so far is always 447 meter, is this fixed or defined by ESRI?

Regards,

Wentao
0 Kudos
WentaoChe
Occasional Contributor II
Clayton,

My new project has 2.56 billion points for DSM, the terrain building failed when using Z-tolerance filter, the error message is Out of Memory, then I tried 2m WindowSize, it works. I hope that I can set the parameter to control the memory used, I can add as much memory as it needs.

I tried several grids created by WindowSize and Z-tolerance, I found that there is no big difference between those grids (DEM), only few cells have different values. I read the related helps of 9.4 again, and got the hints from the guideline. 9.4 help gives me much more detailed hints than 9.3.1.

Regards,

Wentao
0 Kudos
ClaytonCrawford
Esri Contributor
Wentao,

Thanks for the questions and feedback.

Making a DEM:
The different pyramid filter options affect how data gets thinned and generalized. They do not affect the full resolution data. If you are only building terrains in order to create rasters using the full resolution data (TerrainToRaster pyramid resolution level set to 0.0) then it doesn't matter if you use windowsize or z-tolerance or what pyramid resolutions were specified when defining the terrain. Going on this assumption, I recommend using the windowsize filter since it's faster. You can also build the terrain using a much larger windowsize than 2 (e.g., 50), it will be faster. If, on the other hand, you want to rasterize using a thinned pyramid level instead of the full resolution level then you need to start thinking about pyramid type and pyramid level resolutions.

If you choose to use windowsize, and might rasterize on a thinned pyramid level, or view the terrain at different display scales, and your lidar is bare earth, then I think the ZMean option is a good one to use. If you have first return lidar then I recommend ZMax.

You can build a 1 meter DEM with a terrain made from any windowsize. The question is, what pyramid level are you performing the rasterization on? If your lidar is sampled at 1 meter, and you want to make a 1 meter DEM, then it really doesn't matter what pyramid type you use or the pyramid resolution definitions because when you run TerrainToRaster you ought to rasterize using the full resolution data by specifying pyramid resolution level = 0.0. I would not recommend building the 1-meter DEM using a coarser resolution pyramid level. On the other hand, if you wanted to make a 2-meter DEM from 1-meter lidar points, than rasterizing based on a 2-meter windowsize is reasonable. It will run faster, not just because of the larger cellsize, but also because you're processing a thinned point set.

If your terrain's tilesize is always 447 it means you're always saying the point spacing is 1. The tilesize we use depends on the point spacing you declare. So, does all your lidar have a 1 meter nominal point spacing? If so, then this is fine.

Out of memory
The 'out of memory' error you got when building the terrain likely related to data density. The first thing to check is the point spacing used to define the terrain. Is it correct? What is the nominal point spacing of the lidar and is that the value used to define the terrain? Another consideration is breaklines. If you have very densely sampled breaklines these should influence your point spacing estimate.

Are some areas of your terrain sampled more densely than others? If you're building a terrain from data that has different sample densities then you should specify the point spacing of the densest data when defining the terrain. For example, if an urban core is sampled at twice the density of the rest of the data used to build the terrain, specify the point spacing of the urban core. The terrain's tile system needs to be based off the densest data.

If you think your data is of relatively constant density I suggest testing that to make sure. Do this using PointToRaster with your lidar multipoint feature class. Specify an output cellsize that's 4x the point spacing of your lidar. Use the COUNT option as the cell assignment type. Note that the value field doesn't matter when using the COUNT option. If your lidar has a 1-meter point spacing, and you COUNT on a 4 meter cellsize, then you would expect to see, on average, a value of 16 in the resulting raster cells. If this isn't the case, if there's a significantly larger count somewhere, this could explain the problem.

Regards, Clayton
0 Kudos
WentaoChe
Occasional Contributor II
Clayton,

Thank you so much for your detailed advices, all you wrote are what I really want to know. I feel that I begin to understand what terrain is and what factors I should take into account to build the terrain.

After I read your information, I feel that I need to read 9.4 Help more carefully for each issue related to Terrain, such as average distance between points and Tile Size since everything is ALREADY there! I should use Las Point Information tool to know the Average Point Spacing Parameter before starting the terrain building process. Now I know my Out of Memory error is from miss-setting this parameter since our Lidar data is always collected to make sure that we can get at least one point in 1m or 0.5m mesh. After I set Average Point Spacing to 0.5 meter, the tile size is changed to 224 meter. I have also tried the PointToRaster with a 4 meter cellsize and COUNT option, this also helps me a lot to know how much points are located in each cell.

From now, I will try to create my DEM (bare earth) with Z-tolerance filter and full resolution data, and create DSM with 4 times of Average Point Spacing for WindowSize and full resolution data.

Regards, Wentao
0 Kudos
WentaoChe
Occasional Contributor II
Clayton,

In order to know which WindowSize is the best and the fastest for building a DEM terrain with the full resolution, I did a test with WindowSize ranged from 2, 4, 8, 16, 32, 48, 64, 96, and from 30, 40 , 50, 60, 70, 80, 90, 100, the results show 50 meter is a good choice for my data with 6.2 millions points, the difference is within 1 seconds ( time for terrain building) for WindowSize from 40 , 50, 60, 70, 80, 90 to 100.

Just as you pointed out, WindowSize 50 meter is a very good parameter to build a DEM terrain with the full resolution.

Regards,

Wentao
0 Kudos
PeterWilson
Occasional Contributor III
Hi Clayton

I need some advice with regards to generating two terrains for an entire continent (South Africa). Unfortunately we are not as fortunate to have Lidar data for South Africa. We currently only have 20m and 5m contours. I found a white paper by ESRI describing the use of contours to generate terrain datasets, where they recommend specifying the SFType as Mass Points. The 20m contours dataset consists of 150 tiles. I have loaded the tiles separately in order to manage updates that we receive. I have projected the tiles to a custom Albers Equal Area Conic Projection for South Africa. I'm also making use of drainage lines that are hydrologically correct that I'll convert into 3D polylines by intersecting the contours and linearly interpolating inbetween to reinforce the drainage. Please can you offer some advice on the use of Z Tolerance or Window size when using contours to generate a terrain. I would like to use the terrain to extract DEM at full resolution for hydrology analysis.

Regards
0 Kudos