Help with Optimize Raster Tool and Coordinate Systems – Starting a Large Scale Scanned Map Mosaic

[DamonNelton]

Long question, been at this for awhile and could really use some experienced help.

End goal – Build a raster mosaic with thousands of scanned maps that can be published to our Portal environment as image services.

Our Data - Thousands of 3-band 8-bit geo-tiff scanned map images with various coordinate systems but mostly NAD83 State Plane systems that cover most of North America.

Current Workflow – Built a normal raster mosaic with a Web-Mercator coordinate system (not sure this is the correct decision). Our data is located on a local shared file drive, and we are attempting to use the Optimize Raster tool to move the data to an Azure Cloud Store and process the data into MRF JPEG (ESRI suggested that at the scale we are building this we are going to want MRF to prevent performance issues). Once this is complete add the data from the cloud store into the raster mosaic in ArcPro via Add Raster Tool. Once Rasters are added we Calculate Statistics and build Pyramids using the individual GPTs.

Problems – Using the Optimize Raster tool works to create the MRF files and move data to the Azure Cloud Store, but the MRF files lose their coordinate system and images land in the Ocean off the coast of Africa.  --- Note, that if I put our base data (geo-tiff) into the Cloud Store via copy/paste and then add it to the Raster Mosaic it keeps its coordinate-system and lands on the map where it belongs.

Anyone that can help me work through this would be GREATLY APPRECIATED!

[PeterBecker]

Do take a look at Managing scanned maps  workflow. This covers many of the best practices. In short no issue with using WebMercator (assuming your maps are not of the north pole). MRF if a good choice, but best to consider this just as a raster without any projection/georeferencing in  this specific case. It is better to create a table raster that then defines the transforms to be applied to the image. (eg affine, projective or set of control points). Typically you would have a georeferenced the image using a set of control points. Best if these are used to define the transform in the appropriate projection. Note for such scanned maps I recommend against calculating statistics. Not required. Also for overviews, it is often the case with scanned maps that you have other scanned maps with smaller scales that can be used at smaller scales so the number of overviews can be very limited. Also check the doc as it defines how you can set up the footprints and seamlines to clip out the margin/legend data to provide a more seamless map, while still enabling users to access the legend etc if they lock raster to a specific map sheet.

[DamonNelton]

Thanks - You hit on a lot of points

• Managing Scanned Maps Workflow – I have this memorized. It’s great but very high level
• Glad to hear you don’t think WebMercator is an issue
• Overviews – Have had nothing but problems with these. Several hours of professional services later we ultimately decided we don’t need them. Problem solved
• The info in the margins and legends on our scanned maps are very important to us. We don’t want to clip these out. Also, our Mosaic is a non-continues puzzle. A seamless map is not applicable to our use and data-set.
• The MRF part – I’m still very green to this. Why does the coordinate system disappear when converting with the Optimize Raster Tool? Our original data is a bar napkin that we scan on the copy-machine turn into a tiff file and geo-reference in ArcPro using control points making what I call a “Geo Tiff”. Could you provide more details on this MRF and “Table Raster” concept.
• Not Calculating Statistics – I’m not against removing a step but could you elaborate why this is not needed.

[TilmannSteinmetz2]

@DamonNelton - I wonder if you have modified the configuration file for your OptimizeRasters process. We needed to specify a GDAL_translate parameter for the CRS (2193 in our case, see boldface line below): 

<?xml version="1.0" encoding="utf-8"?>

<!--Please note: Paths and extensions defined herein are considered as case-sensitive in OptimizeRasters.py-->

<OptimizeRasters>

  <Description>This template has to be used when you want to optimize and convert your NetCDF imagery to MRF format with LERC compression. The output is in NZTM projection. LERC is especially valuable for higher bit depth data such as newer satellite imagery and elevation models.</Description>

  <Defaults>

    <!--Acceptable modes are [mrf, mrf_jpeg, tif, tif_lzw, tif_jpeg, rasterproxy. The Following items while still supported are no longer encouraged to be used: cachingmrf, clonemrf, splitmrf]-->

    <Mode>mrf</Mode>

    <!--File extensions considered as (Rasters). These files will not be copied from the input path-->

    <RasterFormatFilter>nc</RasterFormatFilter>

    <!--File extensions to ignore completely while copying files/data from the input path-->

    <ExcludeFilter>png,tif,tiff,TIF,TIFF,img,jp2,JP2,IMG,JPG,jpg,sid,SID,tmp,rrd,idx,lrc,mrf_cache,pjp,ppng,pft,pzp,ovr,aux.xml,aux,tfw,TFW,pjg</ExcludeFilter>

    <!--'true' will scan for (Rasters) in sub-directories. Acceptable values are [true, yes, t, 1, y, false, no, f, 0, n]-->

    <IncludeSubdirectories>false</IncludeSubdirectories>

    <!--Compression to use on output (Rasters)-->

    <Compression>LERC</Compression>

    <!--how the data is stored as Pixel interleave or band interleave. For cloneMRF the value should be the same as input (Def: Pixel)-->

    <Interleave></Interleave>

    <!--Compression quality to apply for JPEG compression (Def: 85)-->

    <Quality></Quality>

    <!--LERC precision to apply for LERC compression (Def: 0.5 for int data and 0.001 for float data)-->

    <LERCPrecision>0.001</LERCPrecision>

    <!--Build pyramids? Acceptable values are [true, yes, t, 1, y, false, no, f, 0, n, only, external]-->

    <BuildPyramids>true</BuildPyramids>

    <!--Pyramid levels to create (Def:  2)-->

    <PyramidFactor>2</PyramidFactor>

    <!--Pyramid sampling [nearest,avg,average,gauss,cubic,cubicspline,lanczos,average_mp,average_magphase,mode] (Def: average)-->

    <!--The avg sampling method is only supported by mrf format, which is similar to average, but is a pure averaging of every 2x2 pixels from top left, which is slightly faster -->

    <PyramidSampling>avg</PyramidSampling>

    <!--Pyramid compression [jpeg, lzw, deflate] (Def: jpeg)-->

    <PyramidCompression>LERC</PyramidCompression>

    <!--No data value. If undefined/empty value -a_nodata will not be applied (Def: undefined)-->

    <NoDataValue>-9999</NoDataValue>

    <!--Output tile size-->

    <BlockSize>512</BlockSize>

    <!--This needs to be specified when generating caching or clone MRF, the value should be based on the input raster pyramid factor (Def: 2)-->

    <Scale></Scale>

    <!--If ‘True’ raster output extensions will not be renamed to 'mrf'. Acceptable values are [true, yes, t, 1, y, false, no, f, 0, n]-->

    <KeepExtension>false</KeepExtension>

    <!--Simultaneous threads to use for parallel processing/instances of gdal_translate/gdal_addo/etc (Def: 10)-->

    <Threads>4</Threads>

    <!--Path to save log/completed job files-->

    <LogPath>C:\General\MRF_LERC\Logs</LogPath>

    <!--User GDAL_Translate values. These values will be passed on without any modification-->

    <GDAL_Translate_UserParameters>-a_srs EPSG:2193 -stats</GDAL_Translate_UserParameters>    

    <!--Input cloud storage type to process/download data. Acceptable values are [Amazon, Azure]-->

    <In_Cloud_Type></In_Cloud_Type>

    <!--Output cloud storage type to upload data. Acceptable values are [Amazon, Azure]-->

    <Out_Cloud_Type></Out_Cloud_Type>

    <!--To upload processed data to cloud storage. Acceptable values are [true, yes, t, 1, y, false, no, f, 0, n]-->

    <CloudUpload>false</CloudUpload>

    <!--The Profile name to use for the output S3 credentials-->

    <Out_S3_AWS_ProfileName></Out_S3_AWS_ProfileName>

    <!--AWS Access Key ID to use for the output S3 account-->

    <Out_S3_ID></Out_S3_ID>

    <!--AWS Secret Access Key for the output S3 account-->

    <Out_S3_Secret></Out_S3_Secret>

    <!--Output S3 bucket name-->

    <Out_S3_Bucket></Out_S3_Bucket>

    <!--Output root folder path on S3 to upload converted data-->

    <Out_S3_ParentFolder></Out_S3_ParentFolder>

    <!--Set canned ACL to apply to uploaded files. Acceptable values are [private, public-read, public-read-write, authenticated-read, bucket-owner-read, bucket-owner-full-control]-->

    <Out_S3_ACL></Out_S3_ACL>

    <!--If ‘true’ generated output will be deleted once successfully uploaded to S3. Acceptable values are [true, yes, t, 1, y, false, no, f, 0, n]-->

    <Out_S3_DeleteAfterUpload>true</Out_S3_DeleteAfterUpload>

    <!--The input Profile name to use for S3 credentials-->

    <In_S3_AWS_ProfileName></In_S3_AWS_ProfileName>

    <!--AWS Access Key ID to use for the input S3 account-->

    <In_S3_ID></In_S3_ID>

    <!--AWS Secret Access Key for the input S3 account-->

    <In_S3_Secret></In_S3_Secret>

    <!--Input S3 bucket name-->

    <In_S3_Bucket></In_S3_Bucket>

    <!--Input S3 root folder path to access/download data-->

    <In_S3_ParentFolder></In_S3_ParentFolder>

    <!--The input Profile name to use for Azure credentials-->

    <In_Azure_ProfileName></In_Azure_ProfileName>

    <!--Input Azure account name-->

    <In_Azure_AccountName></In_Azure_AccountName>

    <!--Input Azure primary key-->

    <In_Azure_AccountKey></In_Azure_AccountKey>

    <!--Input Azure container name-->

    <In_Azure_Container></In_Azure_Container>

    <!--Input Azure root folder path to access/download data-->

    <In_Azure_ParentFolder></In_Azure_ParentFolder>

    <!--Set the user access permission level on uploaded files. Acceptable values are ['private', 'blob', 'container']-->

    <Out_Azure_Access></Out_Azure_Access>

    <!--The Profile name to use for output S3 credentials-->

    <Out_Azure_ProfileName></Out_Azure_ProfileName>

    <!--Output Azure account name-->

    <Out_Azure_AccountName></Out_Azure_AccountName>

    <!--Output Azure primary key-->

    <Out_Azure_AccountKey></Out_Azure_AccountKey>

    <!--Output Azure container name-->

    <Out_Azure_Container></Out_Azure_Container>

    <!--Output root folder path on Azure to upload converted data-->

    <Out_Azure_ParentFolder></Out_Azure_ParentFolder>

  </Defaults>

</OptimizeRasters>