Ideas 4 Multi Year Flight Plan (10TB) + Metadata?

[Ravichandran_M_Kaushika]

dear Readers,

thank you for taking the time to help me. 

till recently we had analog imagery captured, geo-rectified and served as a map service on ArcGIS server.  For the last couple of years, we have started using image service for the images with a separate metadata map service based on what the end users wanted to see about the image.

Before the collection begins,  a vector file (shape of file geodatabase) given to vendors to collect imagery.  Imagery would be collected digitally moving forward with all fiducial information.  

Among the information the users currently care about are the actual date of collection,  batch (FY2017 or FY2018 etc), quad name (or DOQQ), and the contract areas unique ID. 

To get an idea about these, I was closely looking at the Multi-year imagery by Bozeman GIS (Bozeman GIS‌)

http://www.arcgis.com/home/item.html?id=b57b656dc5824edcbb40b02f7acad893 

The multi-year imagery service when viewed in ArcGIS Pro had attribute tables that would be present while creating a mosaic dataset.  Along with the 'default' columns from the software, there were collection date and other user-defined columns. these columns were not present when viewed in 'ArcGIS Map Viewer' .

Imagery collected by our agency might be over 10TB as tiff and close to 4 or 6 TB in MRF.   There are an estimated 10,000 to 12,000 raw picture files or contract areas that need to be studied.  This is an annual effort for different parts of the country as determined by the business and legal needs.

Questions:

1.   Should we get 1 mosaic dataset per year or break 1 years image files into many states or region mosaic datasets?   if we break the image collection by state/ can we get a national single mosaic dataset to be served as a service?

conversely, what is the upper limit of mosaic datasets?

2. As indicated before the end users might be interested in seeing certain attributes of the collected image - is it a desirable practice to add 2 or more columns as done by Bozeman GIS‌ in the above service?

or create a metadata map server with 'envelope'/ footprint of the images from the mosaic dataset --> SDE --> ArcGIS map document --> map server.  An approx. max of 12,000 vector records with a few attributes a year would be created -   this layer/service is not worrisome to us.

3. Does ESRI recommend 1 mosaic dataset with (or without) metadata columns per year or have a single mosaic dataset for many years with updating the dataset with a year tag for the each of ~5 TB MRF files/year?  

If people consume the above data in an application and when the application goes to production, there has to be a conscious effort/time spent to ensure the data is 'consistent' with previous years 'style'.  That should be planned up front.

We are thinking out loud and trying to 'adopt' industry's best practices as we move forward for increased efficiency.

thanks for your time again.

regards

Ravi Kaushika.

[PeterBecker]

1.   Should we get 1 mosaic dataset per year or break 1 years image files into many states or region mosaic datasets?   if we break the image collection by state/ can we get a national single mosaic dataset to be served as a service?  conversely, what is the upper limit of mosaic datasets?

How do you currently manage the projects? Probably by year. Suggest you create a mosaic dataset for each project and perform the QC on that mosaic dataset. At this stage also consider adding fields that define additional metadata about each image (eg AcquistionDate). Also consider creating overviews of each project to provide faster access at small scales.  Add similar attributes to these fields. Then create a ‘derived mosaicdataset’. Essentially you create mosaic dataset that contains all the records from all the projects. Then serve this as an image services. For more details see ‘Using Mosaic Datasets to Manage Imagery’ in www.esriurl.com/imageryworkflows

There is no defined upper limit to the size of a mosaic dataset. There are mosaic datasets with 10’s of millions of records. It’s not clear if you are using on-the-fly orthorectification or not. There are a lot of advantages. Also see http://www.arcgis.com/home/group.html?id=b65f2601e0084e32afab3eb488fa8a67&view=list&start=1&num=20&s...

2. As indicated before the end users might be interested in seeing certain attributes of the collected image - is it a desirable practice to add 2 or more columns as done by Bozeman GISin the above service?

In the mosaic dataset you can defined what fields get transmitted to the users. If user has configured popup then the attributes will be displayed. 

3. Does ESRI recommend 1 mosaic dataset with (or without) metadata columns per year or have a single mosaic dataset for many years with updating the dataset with a year tag for the each of ~5 TB MRF files/year?  

As above create by project then create a derived mosaic datasets. This way when new imagery becomes available you just update the image service and all maps/apps continue to work

  If people consume the above data in an application and when the application goes to production, there has to be a conscious effort/time spent to ensure the data is 'consistent' with previous years 'style'.  That should be planned up front.

This is why you want to standardize the list of attributes at the start. When creating the derived mosaic datasets all fields will be copied across.

[Ravichandran_M_Kaushika]

Peter Becker,

good morning. thank you for your pointed response.  I will be sharing it with my colleagues for additional discussion.  I will answer some of your questions:

1. Derived dataset - we 'decided' to merge batch 1, batch 2, and 3 into a 2018 image service for faster display at zoomed out (national extent) - small scales.  When seen on ArcGIS map viewer, we would see only 2 records on the attribute grid.  We decided to serve out metadata from foot print export to a file geo-database/ Feature class to modify the attribute structure and onwards to SDE feature class.   Till recently, there were 'analog' and digital collections - they were merged and served by year.  Again, metadata for all the years were served as a separate map server service.  In 2018, metadata was presented as both map and feature services.
2. We are NOT doing on the fly ortho-rectification; that is one less problem for us.   Vendors would be providing us with 'rectified' imagery.
3. Thanks for mentioning that users need to have pop-up configured for seeing Bozeman GIS kind of services.
4. Thanks for the clarifications about standardized attributes and mentioning that there is no upper limit to the size of the mosaic dataset.

On a side note, I am a Arc Objects (VB macro and .NET), Web ADF, Server ADF, Silverlight, JS, and a web app hard-core vector person.  Apologies if I exhibited my raster ignorance with my questions.

thanks for the pointers and I will follow up on them. 

thanks and regards

Ravi Kaushika.

[CodyBenkelman]

Ravi

All very good questions, and it is good that you are planning ahead!  For a project this big, I am confident that your list of questions is only the beginning.  We have published a great deal of advice and best practices at http://esriurl.com/ImageryWorkflows.  Your use case is what we refer to as Preprocessed Orthos, so look specifically for that topic.

Briefly, I would recommend a "source" mosaic for every individual collection, and then all source mosaics may be combined into one single "derived" mosaic to centralize your entire collection.  (Pay close attention to the advice - always use the Table raster type when adding a mosaic dataset as input to a "derived" mosaic dataset).   

Yes you should add any custom fields necessary to allow you to find key metadata quickly.  I wouldn't hesitate to add 10 or 20 fields - you will not suffer any performance problems by adding a few more - so plan for these metadata fields at the outset and ensure EVERY collection populates those fields before being merged into the centralized "derived" mosaic.  

One decision you'll need to make is whether or not to cache the imagery.  I would probably advise not to cache, but it's hard to say for sure without further information.   If you do not create cache, you won't need to publish the metadata as a separate map service (with 'envelope'/ footprint of the images) - serving the derived mosaic as a dynamic image service provides this capability.  

Cody B.

[Ravichandran_M_Kaushika]

Cody Benkelman,

good morning. thank you for your pointed response.  Between your response and  Peter Becker‌ 's response, we should have good starting point.

We might add less than 10 attribute fields - thanks for clarifying that adding attribute fields would NOT affect the performance. 

We don't plan to cache this service.  So it is good to know that serving the derived mosaic dataset as a dynamic image service (with additional attribute fields ) eliminates the need for a separate metadata service.

thanks for your help and regards

Ravi Kaushika