Update part of a Mosiac Dataset 10.2.2.

NicholasO_Connor · ‎06-02-2015

Hello ESRI Users

Please note names of TIFs and paths are made up for this example.

SDE 10.2.2. / ArcDesktop 10.2.2.

December 2014: I created a colour Orthophoto Mosiac Dataset called AREA1 in SDE 10.2.2

[1] CreateMosaicDataset D:\RASTER\conn\1.sde AREA1 4 BAND 8_BIT_UNSIGNED

[2] AddRastersToMosaicDataset I added 1000 TIFs to the Mosiac Dataset – these are stored on the File System D:\RASTER\tifs\*.tif

[3] DefineOverviews Identified 8 overview levels - These are stored on the File System D:\RASTER\overviews\*. tif

[4] BuildOverviews

In part [2] the tifs are called 1.tif,2.tif, 3.tif…..999.tif, 1000.tif

Users access the Mosiac Dataset via an SDE connection.

All worked ok. Performance was good.

HOWEVER, we have now noticed that 4 tifs need replacing (500.tif,501.tif,502.tif and 504.tif) due to poor image quality.

I have naturally read the ESRI documentation and done some “googling” but I am still 100% unsure of the correct procedure.

Can anyone give me a “cookbook” / urls on what is the best procedure to make this update from your real-world experience?

Much appreciated.

larryzhang · ‎06-02-2015

Nicholas,

Yes, this is very common, when updating the existing/ online MD.

The solution to it can be divided into two:

1. Without re-building the overviews:

The new 4 TIFFs can be added into the existing MD, and then build the footprints on those additions of images only. If those 4 new images have same coverage as the old ones, just delete original 4 images with low quality. In this way, when users zoom-in closely, the good images will display correctly. (However, the original overviews with low quality of images still will be visible at low mapping scales. If it is not acceptable, take the second action as follows).

2. Rebuilding the overviews:

Just delete the overviews from the above MD, and then create /build the new overviews again.

The above workflow wouldn't affect the online image service (which was published from this MD) that applications and users are using , in particular, the first one. If to re-build the overviews, it is better to do after work hours..

GabrielUpchurch1 · ‎06-02-2015

Hi Nicholas,

This is a great question and unfortunately, the available documentation does not do a good job of prescribing an approach for specific situations. This is likely due to the incredible variability in mosaic dataset implementations and the needs/objectives of users. As usual, there are probably a few different ways you could go about the update process and achieve favorable results and in my experience, setting up a test mosaic dataset and working out a proof of concept based on the documentation and trial-and-error testing has been the best approach. What you don't want to do is test anything out on a mosaic dataset that contains 1,000 tiffs and has been configured to your specifications. In other words, you want to have confirmed that the approach you have chosen will actually do what you need it to do for your specific mosaic dataset implementation. Many things you do to mosaic datasets, such as using the Synchronize tool, are not reversible so investing time and effort into testing usually saves you time and effort in the long run.

Based on the info you have provided, your implementation sounds pretty straightforward but I do have a few specific questions:

1. Did you configure the cell size ranges (i.e., MinPS and MaxPS) or allow the software to do it?

2. Did you configure anything in the Define Overviews tool or accept the defaults?

3. Do you intend to update the affected overviews in addition to replacing the source images?

4. Are you OK with removing the old tiffs from the folder location where are the imagery is stored?

NicholasO_Connor · ‎06-02-2015

Hello

Firstly, thank you both (Larry) for the feedback. Greatly appreciated.

1. Did you configure the cell size ranges (i.e., MinPS and MaxPS) or allow the software to do it?

After loading the tifs and building the overviews I used the CalculateCellSizeRanges tool.

2. Did you configure anything in the Define Overviews tool or accept the defaults?

I defined them myself – through trial and error.

3. Do you intend to update the affected overviews in addition to replacing the source images?

Yes.

4. Are you OK with removing the old tiffs from the folder location where are the imagery is stored?

Yes.

larryzhang · ‎06-02-2015

Nicholas,

To avoid errors, advisable to use the GP tool Remove Rasters From Mosaic Dataset—Data Management toolbox | ArcGIS for Professionals , If you decide to select and delete 'old' TIFFs (or Overviews) from MD.

GabrielUpchurch1 · ‎06-02-2015

Thanks for the additional details. Since you did some customizing to various aspects of the mosaic dataset, I would stay away from the Synchronize tool. Something similar to what Larry suggested should meet your needs. Here is a general approach that I think will work but you should definitely test on something other than your production mosaic dataset.

1. Use the Remove Rasters From Mosaic Dataset tool to remove the rasters that need to be replaced. You can use a query in the tool or a selection in the Footprint attribute table to do this.

2. Use the Add Rasters to Mosaic Dataset tool to add the new rasters. I would uncheck the the "Update Cell Size Ranges" option. If you leave this option enabled in the tool, you risk the software recalculating all of your MinPS and MaxPS values for the entire mosaic dataset to something it thinks is best. After the raster are added, use a selection and the Field Calculator in the Footprint attribute table to add the desired MinPS and MaxPS values for the new rasters.

3. Verify the new rasters are displaying to your satisfaction.

4. If the new rasters represent the same extent of the old rasters, then you should be able to run the Build Overviews tool with the "Generate Overveiws" option and the "Regenerate Stale Overview Images Only" option enabled. Be sure to disable "Define Missing Overview Tiles." If the new rasters cover different extents than the old rasters, then you will probably want to remove the existing overviews and Define and Build overviews again.

NicholasO_Connor · ‎06-03-2015

Hello Gabriel and Larry – again thank you for your answers – I think the recipe for my mosaic dataset cookbook is almost ready. Sorry just some follow-up points.

Firstly, some additional background we do a lot of pre-processing of our tifs. So there is no overlapping of images. Each tif is a rectangle, all the same size, same properties and they fit together “like squares on a chessboard”. Hopefully that is a ok analogy.

Ok back to the 4 points. And again not real numbers just for the example.

In our Footprint Table for AREA1 the OBJECTIDs for the original 1000 tifs stored in D:\RASTER\tifs\*.tif go from 1
to 1001.

The OBJECTIDs for the tifs I wish to replace are 500,501,502,503

The OBJECTIDs for the Overviews in the Footprint Table go from 1002 to 40000

[1] When I use the “Remove Rasters From Mosaic Dataset” tool how would I set up the parameters? (CHECK or unchecked).

Mosiac dataset: AREA1

Query Definition: where objectid in(500,501,502,503)

Delete Overview Images: ?

Delete Item Cache: <we do not generate a cache>

Advanced

Update Cell Size Ranges: ?

Update Boundary: ?

Marked affected overviews: ?

Remove Mosaic dataset items: ?

I am a little confused after reading the documentation on how/where the relationship between the original tiffs and the overviews are stored. So that if you delete the original tiffs, the associated overviews are also deleted.

[2] ok – so the new tifs in the Footprint Table have OBJECTIDs 40001, 40002, 40003, 40004

[3] ok – hopefully 😉

[4] run the Build Overviews tool

Mosaic dataset: AREA1

Query definition: Do I need to give the new OBJECTIDs here i.e. 40001, 40002, 40003, 40004?

Define missing overview tiles: unchecked

Generate overviews: CHECK

Overview generation options

Generate missing overview images only: unchecked

Regenerate stale overviews images only: CHECK

Is the above correct?

larryzhang · ‎06-03-2015

To avoid affecting users and applications, better to add new rasters first and then rebuild the footprints of those 4 new additional rasters.

And then, select and delete rasters (primary) as follows:

[1] When I use the “Remove Rasters From Mosaic Dataset” tool, how would I set up the parameters? (CHECK or unchecked). See the attachment

[2]...

[3]...

If all above OK, go to the next step:

[4] select all Overviews under Category, and delete ...

[5] Define Overviews and rebuild ...

Keep in mind,

[1] This general workflow works fine, when you plan to update or append the existing MD..

[2] Please DO NOT delete TIFFs from hard disk until MD is updated successfully...

NicholasO_Connor · ‎06-03-2015

Hello Larry – thank you for the feedback.

As I understand it you suggest the following approach:

[1] add the new rasters first (AddRastersToMosaicDataset)
[2] rebuild the footprints of those 4 new additional rasters.( BuildFootprints_management)
[3] select and delete the 4 “old” rasters (primary) as you have in the graphic (RemoveRastersFromMosaicDataset_management)
[4] select all Overviews under Category, and delete ... (RemoveRastersFromMosaicDataset_management)
[5] Define Overviews and rebuild ...(DefineOverviews_management / BuildOverviews_management)

Is the tool in [2] correct?

larryzhang · ‎06-03-2015

Yes,

That workflow should work for your case, that is, to update MD (in GDB) in order to minimize the impact (of an online image service). Just make sure that any GP operations should work on 'selected' items...