Mosaic Raster Imagery for Multiple Years

[LanceCole]

I have always used multiple mosaic datasets in a file or enterprise geodatabase to store our aerial imagery by reference.  I  create a mosaic dataset for each new flight and users can access the different years by selecting the appropriate dataset.  As I was preparing to add a new dataset, I reviewed the documentation and recommended process.  Apparently, I should be taking this a step further and building a "master mosaic dataset" by adding each years mosaic dataset to this master and making the dataset time aware by adding date information.  Over all this makes sense; however, I have several questions.

1) How is there an advantage when you would need to filter the data to render each year individually?  We do not do any temporal analysis on the data but only display the data for the end user by year.

2) An ESRI video (Managing Imagery Using Mosaic Datasets and Image Services - 43:58) recommended having only one mosaic dataset per geodatabase.  We have never really had any issues with the seven datasets we currently use and are were planning to add an eighth.  Should we expect to see a performance improvement creating a master dataset?

3) The recommended workflow in Creating a mosaic dataset containing raster data from multiple dates indicates that "This design is optimal when you have less than eight rows added to your master mosaic dataset."  It does not go on to explain why or provide an alternative for larger collections.  What are our options as we are reaching this point?

[Todd_Metzler]

My environment:  ArcGIS Enterprise 10.8.1 with image server.

We use a hybrid approach based on many of the points you've made. Sometimes we have multiple MDS in single FGDB and sometimes a single MDS in single FGDB and sometimes a single FGDB with a single MDS that is build from other MDS as the input source.  Like almost everything we do in GIS data management, the end use case dictates how we decide to package those data and create services from it.

For example, we categorize our orthoimagery collections in four overarching areas: Regional, Municipal, Conservation, Coastal.  We have a single FGDB for each overarching category with multiple MDS based on collection year.

For discovery/catalog purposes we have a single FGDB and single MDS including our entire ortho collections.  This FGDB MDS has no overviews and is not for visualization, just for discovery of the tile/tiles that cover a spatial or temporal extent.  This service is also the input to our tile discovery and download application.

The key to make this work:  Attribute your MDS with temporal information and enable time in your services.

[LanceCole]

Todd, Thanks for the quick reply.

I am experimenting with this on our development servers and do not see a difference between multiple MDS, a master MDS using time slider and the same master MDS using definition queries to filter the year.  They are all rendering quickly and with out issue.  I have not attempted to publish the master MDS to our image server to see the performance at that level.

I guess I should have included: ArcGIS Enterprise 10.9 with Image Server.  Mixed Desktop 10.8.1 and ArcGIS Pro 2.8.2 user systems.

[LanceCole]

I found another post on ESRI Community (Mosaic Dataset: Rules for multiple dates ) that @CodyBenkelman states: 

There are some pitfalls in the advice you have been given in this tutorial - e.g. it says "Caution: This design is optimal when you have less than eight rows added to your master mosaic dataset." but unfortunately does not tell you the reasons for this caution, or what alternatives you have. 

I really would like to understand the limitations noted in these posts and videos, but cannot find an specific rational.  We have been provided another 8 image raster sets that we would like to make portions available to our users.  I want to build the mosaic datasets the best way possible for optimal performance and allow for future growth. Our AOI is only about 75 square miles and with the additional rasters noted above, about 2.8TB of data once overviews are generated.

[CodyBenkelman · Accepted solution]

Lance 

Sorry I didn't respond soonr.  The single best resource for these questions about "What is recommended?" is our Imagery Workflows site at http://esriurl.com/ImageryWorkflows.  Specific sections I would recommend are the overview topic of Image Management starting at https://doc.arcgis.com/en/imagery/workflows/best-practices/what-are-best-practices.htm (continue with the rest of the section on Image Management) and a specific discussion of the multi-year case in  https://doc.arcgis.com/en/imagery/workflows/resources/managing-preprocessed-orthophotos.htm 

Regarding that post that you found from 2016 - and I regret that I did not discuss this more deeply at that time.  My caution about the 'pitfalls' focuses on which raster type is used when adding a mosaic dataset (MD) as input to another mosaic dataset.  

Briefly:

• I recommend *against* using the default "Raster Dataset" raster type which is recommended in the post you reference  Creating a mosaic dataset containing raster data from multiple dates  If you have 3 input MDs and use that raster type, your master MD will only have 3 records.  That may seem convenient and in some limited cases I'd say "Yes go ahead" but it can lead to performannce problems if you have 8 or 10 or more MDs and especially if you 'nest' them further in another layer of hierarchy because you've embedded database tables as single records inside other tables.
• What I *do* recommend in most cases is to use the Table raster type.  In my last statement, if you have 3 MDs with 100 records each as source data, your master MD will have all 300 records.  This avoids performance problems but it does (generally) require the additional step of defining one or more custom fields e.g. "Year" or "Date" as you note, and I also recommend adding a field (we typically call it "Dataset_ID") into each source MD so that the master MD will clearly show the source MD for each of the 300 records. 
  • Hoping to avoid confusion, I should note that some of our documentation refers to the master MD as a "Derived" mosaic dataset - but this name is simply a convention, not a formally defined term.
• You also asked "How is there an advantage when you would need to filter the data to render each year individually?"  If you only have a small number of image collections (years or separate projects) this is not a major concern, but if you share each as a separate image service, those services will consume resources on the server even if nobody is using them.  If many collections are shared in a single image service, your use of server resources is more efficient (better performance).  The Landsat service is a good example - millions of individual records and it performs very well.   But yes, you do have to provide users with the ability to select images by date (or other attributes as necessary) - see the Time Selector tool in the Landsat Explorer app.
• Last, re: the Esri video in your 2nd reference, I do not agree that you need to create a new GDB for every MD.  I've never attempted nor felt a need for hundreds of MDs - and there probably is some reasonable limit - but multiple MDs in a single geodatabase should not be a problem.

Cody B.

[LanceCole]

Thanks Cody.  I was hoping you would chime in.  

I see you point now on using a single maser mosaic when hosing services.  I have been testing only running mosaics from our enterprise database directly and not via a service.  Having multiple mosaic datasets individually hosted as a service definitely would consume more resources.

I will look into the "table" raster type.  I have always used the default raster type as our imagery is all standard aerial imagery.  Our imagery varies in size over time.  The 2001 imagery has 440 rows and just the TIFFs are 10.3GB.  The 2021 imagery is 2512 rows for the TIFFs and 240 GB.  However, as you pointed out, look at the Landsat imagery with millions of records.

Ultimately, we had not noted any performance issues with 7 or 8 datasets and the main reason for my questions.  I think we would have started to see issues and why I needed a better understanding pertaining to multiple year datasets.  Thanks again.

[CodyBenkelman]

Lance - always happy to help, and I know it can be challenging to find the best advice.  In some cases our recommendations change over the years as the software matures, and in other cases it's a gray area for what is "best".

One additional point I should have added is that our Imagery Workflows site provides examples of scripts for automating the creation and maintenance of mosaic datasets.  In any configuration with more than a handful of mosaics and a dozen image collections, I strongly advise *automating* as much as possible - among other things it will document how each mosaic was built, populated, and configured, and facilitates re-creating a mosaic if you're testing "will A or B work better?".  

The sample scripts are designed to be customized to meet your needs - there is a bit of a learning curve, but I'm confident it will pay off over the years.  Let me know if you need more info.

Cody