Raster Mosaic Datasets in or out of SDE? Basic Questions...

[JamesMorrison1]

To the Forum Readers.

Sorry about the basic questions to follow but I come more from a Database (Oracle) background than GIS. I have read the ESRI support guides but still don�??t quite get the �??big picture�?�.

Our new planned system Oracle 11.2 / ArcDesktop 10.1 / SDE 10.1 �?? currently Oracle 11.1 / ArcDesktop 9.3.1 / SDE 9.3.1, here we have both Raster Datasets (with Pyramids) and Vector data stored. In the new system we wish to use Raster Mosaic Datasets with Overviews in FGDBs and no longer in SDE.

We have two main type of �??Users�?�: Normal access via ArcMap Online Services.

[1] Is there any reason still to store the Raster Mosaic Datasets in SDE and not FGDBs?

[2] Would it make sense to store the original images on the file system with the overviews and then the Mosaic Dataset (the �??attribute tables�?�) in SDE?

[3] Would it be better to store the Raster Mosaic Datasets in FGDBs and then generate Map Caches for the Users (with ArcMap) / Online Services to access?

Any pointers would be greatly appreciated.

[JeffreySwain]

Primarily I suggest storing the mosaic datasets in SDE, when you are interested in database level security.  Otherwise the file geodatabase is more convenient to handle and the overviews can be easily accessed(they will be in a folder near the file geodatabase by default, within SDE stored in the raster column and not visible via ArcCatalog).  Since you are using Mosaic Datasets, you will not have to 'mosaick' the rasters together prior to adding them to the mosaic dataset and you can keep them in their native format.  Also since you are using 10.1, you can cache the image service directly if you so desire. 

Image Service Patterns

[JamesMorrison1]

Thank you kindly for the replay Jeff.

For clarification: "Also since you are using 10.1, you can cache the image service directly if you so desire."

Does this mean, for exaample, I load the TIFFs we have into a FGDB as a Raster Dataset - then create an image service  - then cache the service?

[larryzhang]

James,

The question is not basic; in fact, it represents some of confusions that also we are facing in practice.

Here we would like to share a few of hands-on experience for reference, which is based on our last 3-year operational practice with the Mosaic Dataset model:

Firstly, as new raster model, the Mosaic Dataset can be efficiently used in practice to manage mostly �??pre-processed�?? rasters as image service via FGDB to meet operational needs, even though FGDB still is not SMART enough right now.

Secondly, for small project, the Raster Dataset model can be used to manage raster images and then served as image service. However, if containing raster images is going to larger (saying, 100 GB or more), the Raster Dataset model should not be used, because it fundamentally �??mosaic�?? all those individual raster images during loading, so that it takes very long time to do, in addition to �??limited�?? raster function capability, compared to the Mosaic Dataset model.

Thirdly, on direct cache on �??larger�?? image service in 10.1 (including via ArcSDE/Oracle), it still is questionable in practice. We expect that ArcGIS Server and Raster teams should investigate it and see if to improve this algorithm as #1 priority �?� (For more at http://forums.arcgis.com/threads/66996-Caching-vs-Image-Service-performance-question?p=250278&viewfu... ).


Regards,

Larry @ Aramco

+++++

Notes:

Personally, users should not manage Mosaic Dataset with overviews still via ArcSDE/Oracle in practice, because it will create many potential headaches, in particular, dealing with massive rasters over network, in addition to the difficulty to 'UPDATE' image service and different security mechanisms between the Oracle/ArcSDE and file system...

On directly managing 'sensor or camera-based' images within the Mosaic Dataset model, hopefully, it will take short time for ESRI to innovate and implement all major imagery processing algorithms, so that users will be able to effectively and directly process major �??sensor-based�?? satellite images to meet operational requirements. In other word, until now (10.1 SP1), users are still not able directly to handle 'sensor or camera-based' images properly because of lacking effective algorithms to deal with color enhancement and accuracy adjustment in Mosaic Dataset, including GeoEye-1, IKONOS-2, or SPOT-5.

Of course, with an improvement within 10.1, users can do something more in the Mosaic Dataset model than 'pre-processed'. For details, read at http://forums.arcgis.com/threads/71343-From-imagery-processing-perspective-the-mosaic-dataset-model-... .

[JeffreySwain]

James,

What you described, creating the mosaic dataset based service and then you can cache that image service without having to add to a map document and then caching as a map service.

In terms of what Larry mentioned, the SDE mosaic dataset works, I but I agree in terms of logistics and then cleaning up the intermediate processes. Since the overviews are not visible via ArcCatalog for SDE mosaic datasets, if you want to remove them or see them directly, you will need to access them directly via the database management software.  Also changing the paths of the overviews do not work as expected with the GUI process as the path is inherent in the geoprocessing tools as the path will not be visible and have to be adjusted other ways. 

There are reasons for using SDE mosaic datasets, but I would recommend trying a smaller section rather than simply go forward to see if it matches your needs.

[JamesMorrison1]

Thank you kindly Larry. Much appreciated to read about your experiences.

Our data is as follows for each Raster Mosaic:

684 TIFFs
Each Tiff 150 MB
3 band, cell size 0.25 x 0.25, unsigned 8 bit
Each TIFF col x row 8750 x 6000
Pre-processed so no overlapping imagery tiles

[1] Do you store your Overviews in the FGDB or directly on the File System?

[2] Our idea is to use Mosaic Datasets with �??just�?� the Attribute Table stored in SDE with pointers to the TIFFS and Overviews on the File System �?? good or bad way?

[3] From your experience (and difficult to answer I know) would you recommend using a combination of Pyramids and Overviews?

[4] Under Define Overviews > Overview Tile Parameters: would you recommend a pixel size and number of bands given the above information?

Sorry for the number of questions.

Cheers
James.

[larryzhang]

James,

Pls try the following recommendations:

[1] The Mosaic Dataset, which will manage all 684 TIFFs from the file system over network (or local SAN), should be created with overviews in FGDB together;[INDENT]
The creation of Overviews associated with Mosaic Dataset in FGDB is mainly for better performance. Certainly, if the display performance is not your big concern (saying, it may be because of small raster datasets in your MD), the attribute �??MaxPS�?? in the Footprint can be increased to very large value from the default, so that all raster images are able to display at all scales in MD. At this case, it is surely not necessary to create Overviews at all (if that is your case, you can also manage Mosaic Dataset via Oracle /ArcSDE). Of course, on our practice, the creation of the overviews in MD is highly desirable.

[/INDENT]


[2] It is not dad idea to use the combination of �??attribute�?? table in SDE and Mosaic Dataset with overviews in FGDB. But, the Mosaic Dataset �??footprint�?? should have a common field with �??attribute�?? table in SDE;[INDENT]
However,the mosaic dataset model and FGDB in 10.1 provides more flexibility to manage attributes via �??Footprint�??. If your image services with attributes will be only used (view , query, analysis) in ESRI applications like ArcGIS, it is good idea to manage all attributes via Mosaic Dataset Footprint in FGDB directly. 

Here, personally, for ESRI, it is advisable to release restrictions on (or improve) �??query�?? capability via FGDB in the future's release of Mosaic Dataset model, which should allow to access/ query attributes of image services over Online Web Services, even other applications like Google Earth (Enterprise).[/INDENT]


[3] On ESRI documentation, the creation of both pyramids and statistics on each raster image (even mosaic dataset) are recommended for performance (and color balance, when dealing with �??sensor-based�?? RAW images). On our practice, however, the mosaic dataset with overviews must be created;

[4] On your case, the following parameters can be used for 'Define Overviews': (a) pixel size - 0.5; (b) number of levels �?? 5 or 7....[INDENT]
If not desirable as expected, you can also try to check the option �??Force Overviews Tiles�?�. Note, the 'derived' overviews with this do not meet our expectations mostly , but maybe OK to you.

[/INDENT]

Hope helpful!

[JamesMorrison1]

Much appreciated Larry - very helpful.