Implement folder structure in Azure Blob Storage for Raster storage

Question

Hi There, As part of our Azure cloud migration, I am looking into using Blob Storage for storing raw raster data and well as storing caches.

As per my current understanding Azure Blob Storage is more of key/value pair and doesn't support nested structure within a container. Esri recommends structuring data folder like this below.

https://doc.arcgis.com/en/imagery/workflows/best-practices/data-structure.htm
DataType\Source\Type\Geography\Date

For example: Satellite\GeoEye\GeoEye1\Europe\2001

I am curious how folks are using Azure Blob for raster data management.

Thanks in advance

AndrewRyder1 · Answer

We are also going down this rabbit hole. currently we have the rasters stored in Azure Blob Storage and we do have a "folder structure" in use. You kinda have to trick the blob storage if you want to generate the structure prior to loading any data. I do this by adding a txt file to each directory as a place holder. This txt file will prevent the dir from disappearing if there are no blobs contained in it. You can name it whatever you like (readme.txt, dir.txt, etc..). I like to include some basic information like the dir structure, contact info and a brief description of the contents. Those details are of course completely optional. I use Azure Storage Explorer and follow these general steps:

* Connect the container
* Click create new folder to create the top level folder
* Immediately after creating the folder upload the .txt file mentioned above.
* Repeat for each dir level you want.

There are more elegant and automated ways to accomplish this, in-fact, you could create the file structure you want, with those place holder files on your local disk and then use AzCopy or Azure Storage Explorer to upload the entire directory structure all at once.

All that said, we are still trying to figure out the ins and outs of storing imagery and point clouds in blob store and actually use it. Interested in any tips or tricks that anybody might have regarding this.

VickyS · Answer

Have you gone down this route? We are also considering blob storage for raster and point cloud data.  If you have any feedback it would be appreciated. Thank you

Sunnywaygis · Answer

I also wanted to mention that we have significant amount of raster data (Approx. 200TB)  and we are very interested in using Azure Blob storage for  cost efficiency compared to Azure Files, and NetApp Shares.