Mosaic dataset statistics and color balancing with MrSID

[MichaelNesius]

Hi All,

I'm hoping to get some advice/help/feedback regarding color balancing (primarily, the prerequisite calculation of statistics) to see if I'm on the right track while I'm waiting on things to run.

I am currently working on a large basemap to be built from  an entire state's worth of aerial imagery (in MrSID), but I am having trouble getting the appropriate statistics generated for the mosaics. The way I have things set up now is I have a mosaic dataset for each county and a derived mosaic dataset for the state-wide mosaic dataset.  I ran the Calculate Statistics tool on all of the source mosaic datasets and the derived mosaic dataset. This went quickly, since I used a skip factor. However, when attempting to apply Color Balancing I got an error message stating that not all of the datasets had statistics.

I dug through the help documentation more and found that I needed to calculate source-raster dataset statistics (rather than mosaic dataset statistics) using the Build Pyramids And Statistics tool. I started this (with the Build Pyramids unchecked since I'm working with wavelet compressed MrSID), but it does not have a Skip Factor parameter and it seems like it'll take several days to run. The Build Pyramids and Statistics tool has been running for about ~4 hours and has made it through 2/3rds of a county.

Does this seem right? Is there a faster way to do this?

thanks in advance,
Mike

[larryzhang1]

Michael,

Maybe, the post/answer at http://forums.arcgis.com/threads/96238-Mosaic-Dataset-w-MrSID-Statistics?p=342585&viewfull=1#post342... would be helpful ...

[MichaelNesius]

Thanks for the response and link Larry.

I'm not sure that it is just an issue of these GP tools not working on MrSID though, as I've been able to perform Color Balancing on a test Mosaic of MrSID.

Unlike the production mosaic I'm working on, though, I created the test mosaic from scratch and activated the 'Calculate Statistics' option in the Add Rasters to Mosaic Tool' (rather than using the Calculate Stistics Tool or Build Pyramids and Statistics tool). Since the Color Balancing worked on this test mosaic, I assume that some form of Calculate Statistics can work on MrSID.

My production mosaic is much more complicated. It is a derived mosaic, with source mosaics and their MrSIDs spanning several years and created by/with different Cameras, software, people...


Do you think it could be an issue of certain versions of MrSID (e.g. lossy/lossless, different generations) or trying to color balance a derived mosaic? Or perhaps, a different kind of error being erroneously reported as an error with statistics (during the color balancing attempts)?

Thanks again,
Mike

[RichardHughes3]

Hey Bud,

I've found that by going into the image properties and the symbology tab.  Choose Percent Clip Stretch.  This will prompt for histogram creation.  These statistics will populate with the MrSID.  None of the other methods I've tried with tools have worked.  This method is my go to and works.

Good Luck

Richard

[MichaelNesius]

Thanks Richard,

I think your Percent Clip tip helps by creating the prerequisite Histograms for Color Balancing

I started a larger test mosaic (~150 images) and it seems like I'm coming up against Larry's comment "with color balancing /correction onto 13 images, it is still challenging for ArcGIS 10.x to 'automatically' correct colors among raster images in MD", as it leads to 'ArcInfo (Not Responding)'...while attempting 1st Order Dodging.


PS. I forgot to mention I'm working with 10.0

[RichardHughes3]

True about the color balancing.  From what I can gather, the various color balancing techniques are very different from one another, and in some cases one will work while the others cannot.  When multiple rasters (over 5) are involved, I would recommend setting them into groups by attribute in the footprint.  There is a block setting in the color balancing menu that will treat all rasters in a group as a single raster and not balance them individually.   This should allow smaller groups to be balanced to larger groups.   

Right now I have all of the Landsat 8 scenes that cover South Carolina.  Since each row was shot hours apart, there are drastic differences between the images.  I realize that Landsat data is WYSIWYG, but the analysis process of the entire states seems difficult when the histograms are shifted.

I have had color balancing crash ArMap ( I don't know why), and I have gotten white blotches (I don't know why).  So far my best case scenario with color balancing is with 3rd Order Dodging on 8 bit unsigned NAIP Imagery.  The 16 bit unsigned Landsat imagery does not play well with color balancing for me.  I did read in a python script this morning where an Esri employee used the color grid balance on a Landsat 8 mosaic dataset.  I will be going with that next time.  Looking through Esri python scripts for mosaic datasets seems to be the best place to find the answers for this stuff.  After spending time learning the theory, process, and what does not work, I just need to know what works so I can move on to the analysis ....

[MichaelNesius]

Thanks for the tips, Richard.

I've recently come across a similar suggestion to use the Merge tool to compress tiles that are logically one item in the mosaic (I'm using preprocessed county datasets that have already been internally balanced) into a single row in the mosaic dataset's attribute table. This is supposed to both speed up the color balancing and maintain their internal coherence in the finished color balanced mosaic.

I'm in the middle of migrating to 10.1, so it'll be a while before I attempt to color balance again.

[larryzhang1]

... as I've been able to perform Color Balancing on a test Mosaic of MrSID.

...

Do you think it could be an issue of certain versions of MrSID (e.g. lossy/lossless, different generations)?

or trying to color balance a derived mosaic?

Or perhaps, a different kind of error being erroneously reported as an error with statistics (during the color balancing attempts)?

...

Mike,

Firstly, on a SID, it is hard to judge if CB function works or not in 10.x. As mentioned in the earlier link, most likely, it is challenging to do CB �??practically�?? with any compressed images, including JP2, SID, and ECW (not just within MD in ArcGIS, but also within any other imagery packages, I guess)

Secondly, we have tested different kinds of SID and got similar conclusion in 10.x (we are going to test 10.2.1 sooner).

Third, to do CB, on our practice, it is usually done within MD (not derived MD);

Lastly, not sure if there is the bug related to statistics on SID (as mentioned in the link)...

pls share, if you have any new updates...