topic Re: Mosaic Dataset Issues in Data Management Questions

Mosaic Dataset Issues

BryanFeger — Mon, 28 Oct 2013 17:31:44 GMT

Hello All:

This is my first post on this forum and I would like to apologize in advance for it being a lengthy one. I also would like to apologize for the fact that my knowledge of GIS is not too vast. So, please bear with me.

Anyways, I am working on a project that requires an extensive amount of GIS work. Basically, the company I work for has been provided GIS data from a couple of different companies and have been asked to do a land impact analysis using their data. One piece of the data that was sent to us was imagery. One of the companies had aerial photography flown in Feb. 2013 of the project site. There is a massive amount of data, as it covers a large landbase. There was so much data that they had to send our company an external hard-drive that contained the information. We then placed the information on our network and sent the hard-drive back to them.

Within the main folder, there are four other folders. Two of the folders appear to contain the individual aerial photos in a .tif format. However, there seems to be other file types within these folders; such as .tif.aux.xml, .tif.xml, .tif.ovr, .rrd, and .aux. The third folder is a geodatabase folder and the final folder is an Overviews folder.

So, my work partner and I compiled all of the information that was provided into a .mxd project using ArcMap 10.2. When we brought the geodatabase file containing the imagery into the .mxd, the only thing we would see is a checkerboard pattern. So, we tried removing it from the .mxd and then brought it back in with no luck; the checkerboard pattern was still there. So, doing what any person does when they need answers to a problem, I googled "checkerboard pattern in ArcMap". I then began to learn about mosaic datasets and determined this is what I am dealing with. So, I have tried following advice in a couple of posts I have seen where people had similar issues. Mostly, I have tried to "repair mosaic dataset paths". I also ran the "analyze mosaic dataset", had some warnings and errors reported, and tried to fix them with no luck at all. Some of the Errors and Warnings that ArcMap shows are pixel type of the raster are outside the allowable range, the footprint of the mosaic dataset contains too many vertices, the overview raster dataset referenced has not been completely processed, and the mosic dataset item references files that are inaccessible.

One thing that I notice about my problem compared to similar problems I have read about is no matter what scale I am at in ArcMap, I cannot see the images. Some people said if they were at a large scale they could see the images, but not at a small scale; and some people had the opposite problem. I cannot see the images at all; just checkerboards. I should say, if I turn on the footprints and boundaries layers I can see those. Just no image. So, this indicates (to my understanding) that the mosaic dataset within the geodatabase file is not referencing the overviews nor the individual images. Am I understanding this correctly? It seems that I need to repair the paths that the data is being read from, but my attempts to do this have failed. So, when I run the "repair mosaic dataset paths" what file am I suppose to reference?

Again, I would like to apologize in advance for the length of this post. This is the first time I have dealt with mosaic datasets, I am quite confused about how they work/how to get them to work, and my bosses are counting on me to get this to work correctly. So, if anyone has any suggestions about how to fix my issue I am all ears and would greatly appreciate your input.

Thank you very much,

Bryan

Re: Mosaic Dataset Issues

NeilAyres — Tue, 29 Oct 2013 06:51:49 GMT

Bryan,
I would start by browsing through your data using ArcCatalog, just to get a feel for what is where.
The extra files along with the tiffs are the geoereferencing, pyramids (rrd) and such like stuff. Shouldn't have to worry about it.
A Mosaic dataset has an icon like the jpeg attached. And if you right click on it you can get the properties page which identifies it as a Mosaic.
There are quite a lot of possibilities with Mosaics. But, as far as I am concerned, the mosaic imagery should draw at any scale. That's what the overviews are for, reduced resolution images to draw at smaller scales.
If you click on the mosaic in ArcCatalog, it opens the data table and you can see names of the images included in the mosaic, column "Name".

Just remembered, what version of Arc are you using? A mosaic created with 10.1, is not viewable (I think) by someone using 10.

Unfortunately, the "Repair paths" tool is a little confusing. And if your data provider has indeed created a mosaic with hard coded paths in it, like "Z:/DataStore/TiffFiles/Project1/" and now the data resides on your system in "M:/ProjectName/Tiffs/", but that's the sort of thing it is looking for.

If you can't come right, you might consider just recreating the mosaic. It is actually not that hard. Create a new mosaic, add imagery into it, build overviews. It gets a little more complex if you have overlapping imagery of different vintages, resolutions.

Good luck,
Neil

Re: Mosaic Dataset Issues

GeospatialTechnology — Tue, 29 Oct 2013 09:10:27 GMT

Have you tried building statistics?

Re: Mosaic Dataset Issues

BryanFeger — Tue, 29 Oct 2013 15:31:06 GMT

Thank you for your replies, I appreciated the quick responses.

yppmark - I tried to build statistics, or at least I believe I did. I went to the "Mosaic Dataset Properties", under the "General" tab I scrolled to statistics, clicked on the "options" dropdown menu to the right of "Statistics", and clicked on "Calculate Statistics". It took quite some time for it to do so, and unfortunately did not solve my problem. Does it sound like I did this correctly?

Neil - To start with, I am using ArcMap 10.2, and I honestly don't know what version of ArcMap the mosaic dataset was created with. That information was not shared with us (or at least me). I believe I have read that a mosaic dataset created with ArcMap 10.1 cannot be used with ArcMap 10. Is it possible that it works the other way around? Meaning, if the mosaic dataset was created using ArcMap 10 and I am trying to use it with ArcMap 10.2, could that be causing the issue? Any thoughts on that?

As far as the "Repair Mosaic Dataset Paths" tool, what folder or file is it supposed to be referencing? Does that path need to reference the folder that contains the .tif files used to build the mosaic dataset, the geodatabase file that contains the mosaic dataset, or the overviews folder that contains the overview files? I guess my confusion is based upon what is suppose to be referencing what. The reading I have done on the internet has led me to believe that the geodatabase file contains the mosaic dataset, which references both the individual images that make up the mosaic dataset and the overviews. In these readings, people make the statement "Check to make sure the files are accessible and that the paths are correct." So, it seems like there are multiple paths that I am suppose to set; however, when I do the "repair mosaic dataset paths" tool it seems like I can only set one path to one folder. So, is there somewhere else I am suppose to be setting file paths? Or am I misunderstanding what I have read?

It seems like it is getting to the point where it may just be easier for me to build a mosaic dataset of our own rather than try to use the one that was provided to us. And, I may end up doing that (even though it is not within our scope of work). The imagery that I have was flown at the same time and (as far as I can tell) with the same equipment, so that should make it a little easier. I guess one thing I will ask, how long do you think it would take for someone that has never made a mosaic dataset before make a new mosaic dataset, add imagery, build overviews, and set the coordinate system so it properly overlays the other data? Is it something a person making a first attempt could do in half a day, a full day, a couple days?

Something else I wanted to ask about is transferring mosaic datasets from one hard-drive to another hard-drive or to a server. The person that taught me how to use ArcGIS informed me that files associated with ArcGIS can be sensitive to being transferred. He told me that "drag and drop" or "copy and paste" can cause headaches if care is not taken. When the data was sent to us, it was on an external hard-drive. My coworker connected the external hard-drive to his computer and then "dragged and dropped" the main folder containing all the information onto our companies network drive. Is it possible that could be part of the problem? If so, is there any way that I can look into that further to determine if that is an issue?

Again, I really appreciate your quick responses and help with this issue.

Bryan

Re: Mosaic Dataset Issues

MichaelVolz — Tue, 29 Oct 2013 15:49:40 GMT

Bryan:

When trying to repair the existing broken mosaic dataset, you could legitimately have multiple paths. You could have one path to a folder containing the raw tif images and you could have another path to a file geodatabase containing the overviews for the mosaic dataset. This is how the mosaic datasets are setup at my organization. Hopefully, you can determine the location of these two paths (if this scenario applies) and repair your mosaic dataset using both paths.

You can try fixing one path at a time and after the fix is complete, just check for broken links (There should be a checkbox for this in the upper left corner of the Repair Mosaic Dataset dialog box).

I hope this information helps you out.

Re: Mosaic Dataset Issues

GeospatialTechnology — Tue, 29 Oct 2013 15:58:56 GMT

It's probably easier to create a new mosaic dataset from scratch:

http://resources.arcgis.com/en/help/main/10.2/index.html#//009t0000003p000000

Re: Mosaic Dataset Issues

MichaelVolz — Tue, 29 Oct 2013 16:03:27 GMT

Bryan:

Do you have other smaller imagery sets that you could use as a pilot to learn the ins and outs of mosaic datasets before you try to tackle the larger task of recreating this specific mosaic dataset?

Re: Mosaic Dataset Issues

BryanFeger — Tue, 29 Oct 2013 19:57:00 GMT

mvolz47 - Unfortunately I do not have a smaller set of data to learn with. Also (unfortunately as well) we need to have this imagery available to us very soon. It is quite a situation I am in, that is for sure. In your earlier post, you confirmed my suspicion that there are multiple paths that need to be set. So, does a person set the paths to the .tif files and the overview files from the "repair mosaic dataset paths" tool?

Thanks,

Bryan

Re: Mosaic Dataset Issues

NeilAyres — Wed, 30 Oct 2013 06:19:35 GMT

The repair paths tool just needs to know the old paths to the data (the original tiff files) vs the new path. Then I would simply rebuild the overviews.
Backward compatibility from 10.2 backwards, shouldn't be a problem.

As regards starting again, depending on how many Gb of data there is, is shouldn't take too long. The lengthiest part will be to build those overview. Give it a shot on just some of the data (select individual tiffs instead of the whole directory structure) first. You will find that it is not too difficult.
Good luck,
N

Re: Mosaic Dataset Issues

MichaelVolz — Wed, 30 Oct 2013 11:54:08 GMT

Bryan:

I would try repairing the tif images first. After pointing the mosaic dataset to the correct location for the tif images on your network, I would run the repair again only checking for broken links which should just be the overviews if you supplied the correct path to the tif images. I would then preview the mosaic dataset in ArcCatalog. You will not see the images when zoomed to full-scale (this is where the overviews are used), but you will see the images if you zoom in to a scale where only 1 image is in the extent. Once you have the images pathed correctly, I would then work on repointing the overviews. If you cannot get the overviews to work, I would recreate them (This might take some time, but it should not be difficult).

Re: Mosaic Dataset Issues

brucesimonson — Thu, 31 Oct 2013 03:56:07 GMT

Disappearing imagery at various scales? This sounds reminiscent of something I encountered a while back.

First though, I would make sure that the mosaic dataset is visible at all scales. Right click the layer in the table of contents, pick "properties" from the bottom of the list, pick the "general" tab, and make sure the radio button for "display at all scales" is selected.

If that's right, here's what I had to do, with an earlier version of ArcMAP (10.0). From an email, I wrote to a colleague at the time:

[INDENT]I was able to figure out how to get rasters to show up, when they seem to go away at small scales. It appears to be a pyramid problem.

Two steps: You have to (1) delete the existing pyramids, and then (2) rebuild them. At least, this worked for me.

To do this, you'll need a tool in the toolbox: under �??Data Management | Raster | Raster Properties�?�, you�??ll find �??Build Pyramids�?�.

Step (1) Delete existing pyramids: Execute this tool, enter the problem raster data set, and then click �??Environments�?�. Choose �??Raster Storage�?�, make sure �??Build pyramids�?� is checked, and, in the Pyramid Levels, enter �??0�?� (zero).

Click okay, and okay, again, to execute the tool.

Step (2) Rebuild the pyramids. After step one is done, run the same tool again, with the same raster, but this time, make sure the �??Build pyramids�?� check box is checked, but leave the pyramid levels blank.

Click okay, and okay, again, to execute the tool.

I know, it�??s weird. First time, with �??0�?� levels means �??delete pyramids�?�; the second time, with �??blank�?� for levels, means, �??rebuild them�?�. [/INDENT]

Maybe this will work for you. Good luck.

Cheers,
-Bruce

Re: Mosaic Dataset Issues

BryanFeger — Thu, 31 Oct 2013 15:03:41 GMT

All - Thank you for all of your replies and suggestions to fix my issue. I greatly appreciate it.

So I messed around with the imagery a little more last night with no luck. If anything, I may have gone backwards. Before, I could zoom in to a large scale and see the black and white checkerboard. After last night, when I zoom to a larger scale than 1:4000 the checkerboard pattern is completely gone, and no image. Before, I could see at least the checkerboard.

So I decided to run the "Analyze Mosaic Dataset" this morning to see what kind of warnings and errors would be reported. I have attached an image of the results. I understand that all of these can definitely be an issue; but, if there is anything in this list that catches someone's eye as the glaring issue I would greatly appreciate someone pointing that out to me.

mvolz47 - So, based on what you are saying, I should see two paths when I run the "Repair Mosaic Dataset Paths"; one for the .tif images and one for the overviews. The problem is, when I run the "Repair Mosaic Dataset Paths" I only see one path, and can only choose one path. On top of that, it doesn't give any indication as to what path it is suppose to be; .tif images or the overviews. So, I don't know how to set the paths for each of these items when it seems like I have to option to set one path. I apologize for my lack of ability to better describe what I am seeing. I hope you understand what I am trying to say. Thank you for the help.

Bruce - Thank you for the suggestion. I may have to give that a try. So, do all mosaic datasets have pyramids? The only reason I ask is I believe I was reading a forum thread last week that someone said they don't use pyramids with their mosaic datasets, or there are times when you should use them and times you should not. If this is the case, how can someone tell if their mosaic dataset has pyramids?

Neil - So, looking at the files that contain the raw images, I have somewhere around 65 GB of images!! I imagine that is going to take a significant amount of time to make my own. Any thoughts on how long this could take? The only reason I am asking is because the office I work in has one ArcMap license, and it needs to be shared. If this is something that could take all day to make, I may put off trying to make my own until the weekend when no one will be using it. Thanks for the help.

Re: Mosaic Dataset Issues

MichaelVolz — Thu, 31 Oct 2013 15:24:51 GMT

Bryan:

You have a lot of issues with your Mosaic Dataset, all of which I have never seen before. I would try to take a few images that you know are adjacent to each other and make a small mosaic dataset from just those images including creating new overviews which I would store in a new file geodatabase. I would then see if the images display correctly at all scales. You could then begin adding more images to this mosaic dataset and then update the overviews to reflect the additional images. I would recommend this approach as a way to learn mosaic datasets since you are a beginner and I would not dive into the deep end of the pool if I did not at least know how to swim.

Re: Mosaic Dataset Issues

BryanFeger — Thu, 31 Oct 2013 16:08:56 GMT

mvolz47 - What is funny about that (at least it is in its own way) is this list is quite a bit shorter than it was before. It seems like you have had some experience working with mosaic datasets. In your opinion, do you think the errors and warnings that I am seeing would have been caused by me copy this mosaic dataset from an external hard-drive from another company, placing the geodatabase with the mosaic dataset in my .mxd, and running the "Repair Mosaic Dataset Paths" multiple times? Or, do you think these would have manifested at the time the mosaic dataset was created?

Also, based on what you have seen, do you believe that I would be better off to try and slowly make my own mosaic dataset? Start with a few of the images, make a mosaic dataset, build the overviews and see if I can get it to work in my .mxd? If it works, I can then start adding more of the images and rebuilding the overviews to incorporate them? I am only asking because the project manager is going to be asking me today if it something I think I can do myself or if we need to start asking people some questions. I need to be able to definitively give an answer of either I can do this, or we need help.

I have two folders that contain the .tif image files, but these aren't the only file types in there. In one of the folders each .tif image has multiple file types associated with it. For example, one of the tif images is named Segment-10_corridor_10682_2836.tif. The other files associated with this .tif image are:

Segment-10_corridor_10682_2836.aux
Segment-10_corridor_10682_2836.rrd
Segment-10_corridor_10682_2836.tif.xml

In the other folder, as an example, I have a file called I5_corridor_fy13_0001_.tif. Associated with this file are:

I5_corridor_fy13_0001_.tfw
I5_corridor_fy13_0001_.tif.aux.xml
I5_corridor_fy13_0001_.tif.ovr
I5_corridor_fy13_0001_.tif.xml

Are these associated files there because they are part of the existing mosaic dataset? Will the associated files have any affect if I read the .tif files directly from these folders for the mosaic dataset I will create? If so, do I need to copy all of the .tif files into a new folder and then start adding the images to a mosaic dataset from the new file?

I apologize for asking all of these questions (of which may seem stupid or trivial) but I need to have a pretty good idea of how much work I am going to have to put into this to make it work. The project manager is going to be asking me these questions and I want to have some better answers than "I don't know" or tell them that it won't be that much work and then it takes me days to get it done.

So again, I apologize for all of the "beginners questions" and thank you all for your help and suggestions.

Bryan

Re: Mosaic Dataset Issues

MichaelVolz — Thu, 31 Oct 2013 17:12:39 GMT

Bryan:

I would create a mosaic dataset with just a few images (5-7) and build the overviews, making sure it is viewable in ArcCatalog and ArcMap at all scales so you know if was built correctly. Then extrapolate the time for that number of images to the total number of images to give your manager an estimate of how long it will take you to rebuild the complete mosaic dataset.

Re: Mosaic Dataset Issues

BryanFeger — Thu, 31 Oct 2013 17:26:37 GMT

mvolz47 - Thank you for the suggestion. My biggest concern at this point is the questions I have about the .tif files and the other files that are associated with them that I asked about in my last post. Can I reference those files directly from the existing folders, or do I need to copy the .tif files into their own folder and then reference them from the new folder?

Bryan

Re: Mosaic Dataset Issues

MichaelVolz — Thu, 31 Oct 2013 17:29:11 GMT

You should just be able to add files from a workspace (directory containing the tif file and the other associated files) and they should all be automatically utilized by the mosaic dataset.

Re: Mosaic Dataset Issues

BryanFeger — Thu, 31 Oct 2013 17:31:33 GMT

Okay, so I can create a new mosaic dataset and reference the .tif files from the existing folders. Sounds like a plan. I will give it a try and see what happens. Thank you very much for all of the help.

Bryan