Enterprise Mosaic Dataset - Modern Storage Options?

[Brownschuh]

Hi there.  Our organization is about to receive a large orthoimagery flight package, roughly 1.5+ TB's of data.  In the past, we would store our imagery datasets on a server hosting our enterprise geodatabases.  But our IT team is giving us grief about the space of this flight; it's way larger than what we've stored before.  There simply is no space on the current architecture to support this new imagery.  They mentioned exploring a NAS option, but admittedly I am not too familiar with this as a hosting solution.  

Doing a bit of research I stumbled upon this: https://doc.arcgis.com/en/imagery/workflows/best-practices/storing-mosaic-datasets.htm and ESRI mentions "one popular pattern is to store the mosaic dataset being used for authoring on a NAS or SAN, then prior to publishing, copy it to the appropriate servers using a directory name that is the same as the original."  But honestly, I am not sure what is meant by that statement ...

What do y'all use for storing large mosaic datasets?

Note we will also need to host this imagery on our internal portal via ArcGIS Server/Image Server.

[BillFox]

Hi @Brownschuh 

are you getting several formats delivered of the same area of interest delivered such as geotiff, lidar, slpk?

[Brownschuh]

It is a single area of interest and some kind of image file (I need to confirm with my colleague and the vendor on the final file type they will deliver).  Edit: they are TIF files.

[JoëlHempenius3]

I would use a cloud storage in this case.

ArcGIS Server and Pro support several cloud storage providers. https://enterprise.arcgis.com/en/portal/latest/use/data-store-items.htm 

You just put it there once, make a mosaic if needed and you can publish it as an image service to the ArcGIS Server from Pro Share a web imagery layer—ArcGIS Pro | Documentation 

I personally have done this with Azure Blob storage:

1. Copy all the files to Azure blob using the Azure CLI 
2. Create a mosaic in blob storage
3. Load all the indivual image files into the mosaic
4. Publish the mosaic as image service

[Brownschuh]

Glad you brought up the cloud.  I had mentioned Azure as I had briefly dabbled with it at a previous job (not in the context of storing rasters).  And since we are a Microsoft shop I thought it might be something to pitch as a solution, but they were reluctant.  Something about some fear of backups?  Admittedly, there does seem to be a fear of approaching cloud based solutions for some reason ...

[JoëlHempenius3]

You will have to think about backups in both cases. Sure, you can order a nice NAS/SAN from Dell, Synology or any other vendor, but RAID isn't backup, so if you're serious about your data, you have to buy a second SAN/NAS and put this in another datacenter, and preferably have a cold copy of your data at a third location as well. Is your IT able to support this?

For Azure, backup strategy and redundancy is something your configure when creating the storage. You have some options there. I still would recommend keeping a cold copy of your data elsewhere, because the cloud is just somebody else's computer and stuff can still go wrong.

[Brownschuh]

Thank you for the info.  We will definitely keep a cold copy on hand, but as for a backup that's higher availability, that's a discussion my GIS colleagues and I will have to have with our IT team.

One note, we typically had our imagery hosted on the database server for both convenience and for performance.  Especially for hosting the data via ArcGIS Server/Image Server.  Is that going to be hindered going either the NAS/SAN or Azure route?

[JoëlHempenius3]

No, publishing data from a cloud source is well supported.

https://pro.arcgis.com/en/pro-app/latest/help/sharing/overview/web-image-layer.htm#ESRI_SECTION1_53A...

Just make sure you do it referenced in this case:
Share by referencing the data.

• You can register the folder containing the ArcGIS cloud storage connection file (.acs) as a data store in ArcGIS Pro.
• You can register a cloud store in ArcGIS Server Manager.
• You can register your cloud location as a data store item in the ArcGIS Enterprise portal.

I would go for the third option, because I'm 99% sure under the hood it will also create the registry on the ArcGIS Server

[BillFox]

and network bandwidth, storage space / access speed / cpu & gpu compute resources to grind out the Batch Build Statistics, Batch Build Pyramids, seam lines, boundary, foot prints, overviews, etc.

if these are DEM/DSM geotiff files,  the cache generation chores too