Advice for Mosaic Dataset maintenance & image service publishing

[KeonMonroe]

Currently I'm tasked with publishing an image service from drone imagery and I've been trying to learn more about managing the mosaic dataset object, and also understand if I'm publishing the best image service possible (overall purpose is to serve custom basemaps at street-level scale).

I'm using an Arc Image Server to publish the image service, which is then used to create a custom basemap item in AGOL. 

Here's a breakdown  -

1. Drone team notifies me that the image service needs to be republished, meaning they have finalized adding or removing rasters in the mosaic dataset. They use SiteScan, then create a georeferenced orhtomosaic that is loaded into one large mosaic dataset (stored on a registered cloud store).
2. I review the MD , analyze it, and attempt to republish an updated service.When analyzing the mosaic dataset, I come across several critical errors (see screenshot). Now, I'm trying to come up with a workflow to update the MD. My goal is to identify the necessary geoprocessing tools to best optimize the mosaic dataset and resulting image service.
   Updating Mosaic Data Steps: Everything is green has yet to run. Just trying to plan ahead...
   • Remove Rasters from MD - Ran this to remove "orphaned overviews" (error - The overview mosaic dataset item may not have any participating raster datasets). This process took 36 hours to run... I figured this was fairly critical error to address because they may have removed some rasters from the MD.
   • Calculate Statistics - also seems like this will take a while…
   • Build Footprints (refine footprints and NoData)
   • Generate Seamlines
   • Build Overviews (necessary for service side caching on large collection, bolstering performance)

Here's how the image service configured currently - 

• Service references source data directly from cloud store. I know this isn't ideal but given the size of the mosaic dataset, generating caching to serve imagery would require too much storage (estimated 22 TB at needed scales).
  • I assumed building overviews on a mosaic dataset served a similar purpose as caching on the object's side. Overviews are built on the MD, generating a fairly large amount of data also (albeit manageable). 
  • I also considered separating the collection into several mosaic datasets for more efficient processing...  As mentioned above,  many of the processes above are taking a really long time to execute (running on a new server).

Any general guidance or feedback would be helpful and appreciated. I feel like there's a lot I could improve on  here to get the best image service. Thank you. 

[GordonSumerling]

Hello Keon,

Interesting thing I see here is that your RPAS team is having to notify of the new drone imagery before you do all the work. I was interested to know if they knew about the capability to publish directly from Site Scan to ArcGIS Enterprise.  This capbility has recently been introduced this year and allows Site Scan to communicate directly with your ArcGIS Enterprise or ArcGIS online sites. All your SIte Scan data, not just the Orthomosaic can be published to ArcGIS Enterprise. Full details on this are found in the Publishing from Site Scan to to ArcGIS Online and ArcGIS Entrprise Blog

Hopefgully this helps

Gordon

[KeonMonroe]

Thanks Gordon. We will look into publishing directly from Site Scan. I am still interested if there any recommendations or documented workflows/scripts for handling large mosaic datasets after add/delete rasters.

[CodyBenkelman]

Keon 

This won't immediately answer your specific questions but the single best resource for questions about "What is recommended?" is our Imagery Workflows site at http://esriurl.com/ImageryWorkflows.  Specific sections I would recommend are the overview topic of Image Management starting at https://doc.arcgis.com/en/imagery/workflows/best-practices/what-are-best-practices.htm (continue with the rest of the section on Image Management) and this specific discussion addressing a multi-year case which applies to your issue of adding/maintaining a service with multiple projects:  https://doc.arcgis.com/en/imagery/workflows/resources/managing-preprocessed-orthophotos.htm 

You'll also want to look at specific advice for ensuring efficient access to imagery stored in the cloud - see https://pro.arcgis.com/en/pro-app/latest/help/projects/connect-to-cloud-stores.htm and https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/create-cloud-storage-connect... 

Note that this may have performance delays if your server is not in the cloud and in the same region as the data storage. 

You'll also want to consider how best to manage your overviews, specifically, it would be best for performance if those are stored locally on the server vs. writing them out to cloud storage.  They'll presumably need to be rebuilt each time you add new data to your composite service.

One other tidbit is that your orthomosaics should not need statistics to be calculated.  If you set the MD property to "preprocessed" that tells the software that the image histogram should already be optimized (no brightness/contrast stretch necessary to adjust the display).   If that proves to be untrue (eg. in shadow areas?) you can also configure your image service to enable "DRA" (dynamic range adjustment) which will change brightness/contrast dynamically based on what's currently on screen.  

I know that's a lot to digest 🙂

Let us know if you have more questions

Cody B. 

p.s. I should have noted that Gordon's advice for publishing directly from Site Scan can be a very easy workflow - but just be aware it creates a new image service for each dataset, and does not yet support "Add new imagery to existing service".  Since you want/need to maintain a single image service with all collections, that method of publishing directly from SIte Scan may be very helpful for initial QC but you'll want to continue along your current lines of managing multiple projects in a single master mosaic dataset.