OptimizeRasters; 1m and 2m lidar data LERC/MRF

773
5
Jump to solution
09-08-2017 09:59 AM
Labels (1)
Ravichandran_M_Kaushika
Occasional Contributor

dear Readers,

One of the tasks we are trying to accomplish is to serve lidar DEM (raster) data stored on our servers; lidar collected acrossmany states.   LERC and MRF was a suggested path and we are in the process of evaluating it with a large sample dataset.

we are planning to do the following - please feel free to provide suggestions / recommend changes to optimize our available resources. 

Objective: Evaluate and confirm that LERC/MRF OptimizeRasters processed lidar data would be efficient in storage and transmission.

1. based on the reading, a tolerance of 10 cm of allowable error would yield a reduction of 40% or more.  will that LERC compression consume extra processing resources and additional time for while serving out data?

2. IS Ten cm  too much for lidar 1  m data? will there beerror prone ?.

3a.  Two of our dev servers are running ArcGIS server based services and are arccatalog toolbos functions;  does Optimize rasters toolbox take a lot of processing and memory resources? 

3b.  When I selected a whole folder of Lidar 1m data, the Optimize Rasters  failed half way through the processing - restarting a new instance locks up the machine.  Is there a batch process technique to make use of Optimize Raster tool box?

4. After creating MRF / LERC DEM from OptimizedRasters , we are planning create 2 sets of services exposing the same data; One Optimized and the other with raw data to confirm that we can continue to use Optimized Rasters module from ESRI (JPL, NASA... GDAL etc).

5. With 2m lidar data, 10 cm 'error tolerance' to optimize the size of the file should be fine; for 1m lidar data, any suggestions for the suggested tolerance for optimum storage.

6. there has been statements that question using LERC /MRF efficiencies over unprocessed data due to the compression and de-compression that needs to be done.  Any informed opinion would be useful for us to choose our path.

7. Size of data: our current test data is to the order of 10.1 TB - 1m lidar data for a few counties.  We have users  across the nation and we are actively involved with other federal, state, and local agencies collecting data.  so 10 TB might end up being a fraction of the total data size.

Given the large size of data that we need to share, we decided to reach out for ideas.

thank you for taking the time to read through the entire content.

regards

Ravi Kaushika.

0 Kudos
1 Solution

Accepted Solutions
AndrewWhitman1
Esri Contributor

Greetings Ravichandran, 

I appreciate you updating us with the results of this workflow! I am glad to hear things are moving in the right direction here. 

7+TB of data is a lot to process so I think that breaking this data up into sections for building overviews is a good idea. 

Another factor that affects the processing of overviews is the resolution of the imagery. If all of the imagery is 1m, then this is a non-issue. However, if you have imagery with resolutions ranging from 10cm to 1m, it could be worthwhile to resample the imagery. 

I remember discussing these compression methods with you and the image service capabilities, but if you reach a point where you are receiving specific errors within ArcGIS Desktop/ArcGIS Pro, feel free to reach out to Esri Support Services and create a support ticket. 

The first thing we will likely request is a small sample of the data. 

Wishing you success in your raster imagery journey! 

-Andrew 

View solution in original post

0 Kudos
5 Replies
AndrewWhitman1
Esri Contributor

Greetings Ravi, 

I found some resources that may be helpful regarding MRF and LERC. 

Esri and NASA collaborate

Storing large volumes of data in the cloud

It looks like the LERC compression method is supported for image services as well. 

Tiling Scheme methods

"LERC—Limited Error Raster Compression (LERC) is an efficient lossy compression method recommended for single-band or elevation data with a large pixel depth, such as float, 32-bit, 16-bit, or 12-bit data. LERC compresses 5 to 10 times better and 5 to 10 times faster than LZ77 for float data. LERC is also better with integer data. When using integer data, and the error limit specified is 0.99 or less, LERC is considered a lossless compression."

GitHub

This is an Open Source tool. 

-Andrew

Ravichandran_M_Kaushika
Occasional Contributor

Andrew,

good afternoon.  I was busy with other tasks and did nt see your reply soon enough.

thanks for the links.  I will look into them.

regards

ravi/

0 Kudos
Ravichandran_M_Kaushika
Occasional Contributor

good afternoon. Sorry for the delay in responding. I was taking care of higher priority projects before I could get back to it.  Updates:

1. One of our main servers crashed and that was causing problems during the process of building overviews; we waited for a new server to be ready.

2. on a 10.5.1 machine, we created a file geodatabase, added rasters, defined and built overviews; since ESRI had not suggested an upper limit, we tried to build overviews on all the 7.4 TB of lidar 1m MRF files.

3. it kept failing building overviews resulting in a lot of gaps. to make it easy, we decided to add 1 raster folder (approx. 500 GB) and build overviews.  it has been fairly stable.

4. once the overviews are built, we are planning to make it available as an imagery service,

things have been moving slowly in the right direction.

thanks and regards

ravi.

AndrewWhitman1
Esri Contributor

Greetings Ravichandran, 

I appreciate you updating us with the results of this workflow! I am glad to hear things are moving in the right direction here. 

7+TB of data is a lot to process so I think that breaking this data up into sections for building overviews is a good idea. 

Another factor that affects the processing of overviews is the resolution of the imagery. If all of the imagery is 1m, then this is a non-issue. However, if you have imagery with resolutions ranging from 10cm to 1m, it could be worthwhile to resample the imagery. 

I remember discussing these compression methods with you and the image service capabilities, but if you reach a point where you are receiving specific errors within ArcGIS Desktop/ArcGIS Pro, feel free to reach out to Esri Support Services and create a support ticket. 

The first thing we will likely request is a small sample of the data. 

Wishing you success in your raster imagery journey! 

-Andrew 

View solution in original post

0 Kudos
Ravichandran_M_Kaushika
Occasional Contributor

Thanks for the kind words. One of the things that helped to reduce overview errors were using 'Exclude Duplicates' while adding raster imagery to mosaic dataset.  Usually I see a 'Removed 0 duplicate mosaic dataset items in the log output of 'Add Rasters to Mosaic Dataset' operation,

0 Kudos