topic Re: Creating median, StDev, max, min, count grids from las file with 300 million poin in Data Management Questions

Creating median, StDev, max, min, count grids from las file with 300 million points

AndyRitchie — Wed, 30 Oct 2013 02:21:35 GMT

I am trying to figure out if ArcGIS is even worth working with to process my point data, or if I should look for another solution and just work with the finished rasters in Arc.

I am generating 1.5 billion point clouds of a 30 km strip of river every few weeks. I chop the river into 5 chunks for processing. I have noise issues with specular reflection from the water (using SfM to generate pointclouds), so I have a mess where the water is. The best way to deal with the mess.. I think.. is to (1) clean up spikes with a median filter, and (2) eliminate areas of high noise/low returns with a combination of statistics. So ideally I would be able to generate a grid for each of the following in the .las file - mean, max, min, count, and stdev.

My .las file stores everything as 1st (or last) return (I forget), and basically only has xyzrgb for each point. I tried to convert a <300 million point cloud to multipoint (actually I am still trying and it's been hours), but I am wondering if I should give up. I can get just about everything (except median, which is actually really important) from lastools, and I think I'll be able to get median, and maybe everything, with GRASS. I'd love to use ArcMap, but I am at a loss where to even start. .las file is 8GB.

Re: Creating median, StDev, max, min, count grids from las file with 300 million poin

larryzhang1 — Wed, 30 Oct 2013 08:09:03 GMT

... figure out if ArcGIS is even worth working with to process my point data, or if I should look for another solution and just work with the finished rasters in Arc.

I am generating 1.5 billion point clouds of a 30 km strip of river every few weeks. I chop the river into 5 chunks for processing. I have noise issues with specular reflection from the water (using SfM to generate pointclouds), so I have a mess where the water is. The best way to deal with the mess.. I think.. is to (1) clean up spikes with a median filter, and (2) eliminate areas of high noise/low returns with a combination of statistics. So ideally I would be able to generate a grid for each of the following in the .las file - mean, max, min, count, and stdev.

My .las file stores everything as 1st (or last) return (I forget), and basically only has xyzrgb for each point. I tried to convert a <300 million point cloud to multipoint (actually I am still trying and it's been hours), but I am wondering if I should give up. I can get just about everything (except median, which is actually really important) from lastools, and I think I'll be able to get median, and maybe everything, with GRASS. I'd love to use ArcMap, but I am at a loss where to even start. .las file is 8GB.

Andy,

Certainly, ArcGIS 10.x can be used for some LAS tasks, in particular, as an integrated platform. So, it is better to read http://resources.arcgis.com/en/help/main/10.2/index.html#//015w00000069000000 to judge if it meets your needs�?�

++++++++++
Personally, as LAS practitioners in operation to deal with massive/ RAW LAS dataset for �??complete�?? LAS solution frequently (including LAS processing and DEM/city model extraction), it is worth for corporate organizations to invest time/money onto other vendors�?? LAS solution, like ERDAS/ ENVI/ Leica (XPro) or TerraScan.

Re: Creating median, StDev, max, min, count grids from las file with 300 million poin

AndyRitchie — Wed, 30 Oct 2013 18:14:44 GMT

... it is worth for corporate organizations to invest time/money onto other vendors�?? LAS solution, like ERDAS/ ENVI or Leica (XPro).

Thanks for the quick reply!

Unfortunately I'm not free to spent ~$10-$20K on other software, especially given that I am not using LiDAR-derived LAS pointclouds, and nothing works quite the same with SfM clouds. I have demo'd QT modeler and am in the process of demoing i-site studio, but haven't found an "ideal" tool yet. cloudcompare can handle 300-400 million points no problem but doesn't have editing tools. Lastools is useful but doesn't include a median filter. I am trying GRASS now as ArcMap chugs through generating stats on the multipoint I exported from Lidar. Would be nice if I didn't have to go through the step of exporting to multipoint though since arc now kind of natively supports lidar data.

I should note that I was able to generate a 300 million point multipoint file with ArcMap (about 2 hrs), and it appears that I can generate some stats with the point statistics tool (median and minimum statistics failed, generating grids with pixel depth of 1 bit and apparently all nodata, but std worked). It looks like it takes about 15 minutes per stat run, and I am not clear if when I specify a cell size of 0.5 and a rectangular neighborhood of 1x1 if I am just analyzing the points in that cell, which is my goal (essentially like doing point-to-raster but allows me to calculate median).

Re: Creating median, StDev, max, min, count grids from las file with 300 million poin

larryzhang1 — Thu, 31 Oct 2013 03:56:55 GMT

.... I'm not free to spent ~$10-$20K on other software, especially given that I am not using LiDAR-derived LAS pointclouds, and nothing works quite the same with SfM clouds. I have demo'd QT modeler and am in the process of demoing i-site studio, but haven't found an "ideal" tool yet. cloudcompare can handle 300-400 million points no problem but doesn't have editing tools. Lastools is useful but doesn't include a median filter. I am trying GRASS now as ArcMap chugs through generating stats on the multipoint I exported from Lidar. Would be nice if I didn't have to go through the step of exporting to multipoint though since arc now kind of natively supports lidar data. ....not clear if when I specify a cell size of 0.5 and a rectangular neighborhood of 1x1 if I am just analyzing the points in that cell, which is my goal (essentially like doing point-to-raster but allows me to calculate median).

Andy,

If you have to stay with ArcGIS (plus some of free packages) to minimize mess (noise) in LAS, it is also good to read the 3D blog at http://blogs.esri.com/esri/arcgis/2009/09/02/lidar-solutions-in-arcgis_part7-minimizing-noise-from-lidar-for-contouring-and-slope-analysis/ , which might be helpful somehow, (even a little older).

Pls keep sharing your experience with us (saying, efficiency, accuracy improvement, etc.), if possible.

Regards,

Re: Creating median, StDev, max, min, count grids from las file with 300 million poin

AndyRitchie — Thu, 31 Oct 2013 17:24:35 GMT

Here's my experience so far, working with a .las file exported from Agisoft Photoscan and cleaned with las2las because of an import bug ArcMap has with VLRs in the LAS GeoKey:

I initially imported a ~300 million point LAS file, then generated a multipoint (LAS to multipoint) and attempted to run point statistics with a cell size of 0.5m and a 1x1 rectangular neigborhood for each of median, mean, min, max, std, range, variety. Std and mean were the only statistics that successfully generated. Min,max,median all failed with 1 bit rasters and all nodata. range and variety also gave me 1-bit rasters, but range had 0 in cells that gave me a STD < ~0.4 and 1 with STD >0.4, and variety gave me 0 in all cells with data and 1 only in nodata cells from the original pointcloud (also nodata in STD raster).

point to raster gave me the same result for std and mean as point statistics with a 1x1 neighborhood, and I was able to generate min, max, and range with that tool (also can do std), and it takes about 2 minutes/raster instead of 20. Unfortunately, point to raster doesn't generate median, so I am still striking out on the most important function I am trying to perform from a noise-removal standpoint.

I figured I'd see if size was an issue, so I clipped a small (9 million point) area of the original raster and attempted median statistics on it. That still failed (produced a 1-bit depth raster with all nodata), so I clipped it down to 780 000 points, still failed (same way). I also tried to min and max, and I tried with 1 and 3 cell rectangular neighborhoods. All have same results. 1 bit pixel depth all nodata output. STD still produces a valid raster.