Best practice for storing LiDAR other Raster datasets

[JessicaKirby]

Our office is considering acquiring LiDAR and high resolution Oblique imagery for a few project areas around the state (1-2 TB worth).  I am not familiar with the best practice for storing Raster, LASS or ASCII files or their raster bi-products (terrians, DE, contours).  A colleague of mine mentioned that it is suicidal to store Raster data, let alone LiDAR, on my SDE and suggested that we purchase an image server.  Currently our GIS shop dose not do large volumes of Raster analysis but we may be headed in that direction if the LiDAR is well received. So in order to purchase an image server I will need compelling evidence that it will provide the best possible solution for our minimal needs.

Can anyone give me some best practice advise for storing Raster/LiDAR on an image server vs SDE vs FGDB?

Many thanks!

[CodyBenkelman]

Can anyone give me some best practice advise for storing Raster/LiDAR on an image server vs SDE vs FGDB?

Jessica -  I'll try to keep this brief, then fill in more detail if you or others are interested.

In my view, "best practices" implies a detailed discussion, including not only "what do I do, and why?" but also "how?".  We are currently addressing this by documenting recommended workflows for a variety of types of imagery.  You can see an introduction here http://esriurl.com/6550 which includes a link to the "Image Management Guidebook" in the Help system with detailed discussion.  The chapter on elevation is generally applicable to lidar, although a specific discussion of lidar is not yet included - that will be available *soon*. 

To provide a very high level answer to your questions:

• In most cases, I would agree you should not store lidar or imagery in a database.  There are exceptions in some organizations, but my view is that a key intent of using a database is to manage changes to your data, and that is generally not applicable to imagery or lidar, although it can be VERY important for feature data.




• For managing and serving the lidar or imagery, you should use a Mosaic Dataset.  The Mosaic Dataset (MD) is stored in your GDB, then it simply accesses the source data which remains as simple files on disk (ideally tiled tiff for imagery, tiled LAS files under ~ 200 MB for lidar).




• Note that the MD was developed to manage nadir imagery, but the catalog capabilities of the MD provide powerful capabilities for managing and accessing oblique imagery.  Oblique images will normally be accessed one image at a time, to view an area of interest.




• Specifically regarding lidar, my recommendations depend on what your users need.  Most users ultimately need an elevation surface or other datasets (e.g. contours - what you called "by-products" above) derived from lidar rather than the 3D points themselves.  [For the latter, see below].  For these users, I recommend you store the DEM (bare earth) and possibly DSM (first return surface, e.g. trees and buildings, if needed to determine viewshed or line of sight) as TIFF files on disk, then manage and share them using a Mosaic Dataset in a FGDB.  With the MD, you can easily and instantly generate other products (hillshade, slope, etc.) on-the-fly, for any area of interest and resolution, without storing additional data on disk.  The URL above (and links from that page) provides specific workflows for this. 




• One caveat, since you mention that you don't currently store much imagery, is that the DEM and DSM will require large data volumes - this is 32 bit floating point data - but it is typically 15x to 30x LESS data volume than the lidar data in LAS format, and we have a new format coming soon that can reduce the data volume further.




• Finally, if you have end users who need access to the original lidar data in LAS format for 3D analysis, ArcGIS Server can easily provide an efficient download capability to access limited areas of interest - but in this case you have to store the lidar data on a network accessible disk near the server.  I've done a lot of work with cloud storage, and for infrequent data access, it can provide a cost effective solution.




• Since you mentioned ASCII, I would add that I don't know of any reason for long term storage of data in ASCII format.  It will consume much more disk space, and provide much poorer performance.  ASCII data should be converted (to LAS if lidar, to TIFF in the case of a raster).

I hope this is helpful.  If you need more detailed information, please start with the link above, but feel free to contact me.

Cody Benkelman

[larryzhang1]

...

Note that the MD was developed to manage nadir imagery, but the catalog capabilities of the MD provide powerful capabilities for managing and accessing oblique imagery.  Oblique images will normally be accessed one image at a time, to view an area of interest.

...

Cody Benkelman

Thx, Cody,

More enquiries on �??oblique�?? photos for you to help.

From ESRI raster /3D perspective, with �??oblique photos�?? (and other 3D raster), can you share any successful practice/ projects, which are managed at server and then deployed with ESRI 3D GIS platform?

In other word, pls make more recommendation on server-side management of �??oblique�?? photos data /models (and also 360⁰ panoramas, �??cloud point�?? from stereo-pair or LIDAR, and DSM models from stereo-pair, If possible), so that those 3D raster can /will be easily integrated into ESRI 3D GIS platform (desktop 3D, web 3D), including ArcGlobe, ArcGIS Explorer (3D mode) & web (globe)?


Regards,


+++++++

One of the pre-conditions to manage 3D raster at server is: those 3D data/models should /can be, at least, directly visualized (in 3D mode) at clients. Certainly, it would be better to anayze, measure, even extract 3D GIS data/ models from those 3D raster.

In the market, there are several solution products available, which can be used to manage 3D raster at server, but it looks that those solutions still are very far from seamless integration into ESRI 3D GIS platform (or via plug-in & API).