topic Re: Mosaic dataset operator versus cell statistics - What is wrong ?? in Geoprocessing Questions

Mosaic dataset operator versus cell statistics - What is wrong ??

AlbertoAloe — Wed, 06 Dec 2017 17:08:08 GMT

Hi guys.

I have the following problem (probably due to my lack of confidence when using the mosaic dataset):

I have a set of 366 rasters (one for each day in 2008) with the following specs:

1000 columns, 950 rows
1 band
Cell size is 5 km
TIF format
32 bit floating point
Coordinate system is ETRS_1989_LAEA

The question is :

Why am I getting different results when calculating mean with spatial analyst cell statistics versus a mosaic dataset loaded with the same set of rasters ? There may be a difference of almost 2%

Cell statistics seems to be the correct one because I'm getting the same values in other ways. Rasters are coming from a big netcdf and I verified the avg values of the mean of the same slices using numpy (as well as other tools like sample and multi dimension ones..)

I'm using ArcGIS 10.5.1 on Windows 8.1 workstation with 16 GB of RAM and I've been very carefull to set up the mosaic dataset with the correct properties(pixel depth of the mosaic is 32 bit and maximum size of requests are properly set). I also created a mosaic dataset based on the original netcdf (9131 temporal steps) getting the same results of the tif based mosaic when making a query definition for 2008 year and settimg operator to mean. Values for the rasters are in range 0 - 10000 more or less

Thanks a lot to whoever is able to give me a hint

Alberto

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Wed, 06 Dec 2017 22:45:41 GMT

how do the various approaches handle nodata areas and masks? Cell statistics allows you to explicitly set that behaviour when you run it and in the Environments section (bottom of the help), it enables you to specify how it handles a mask. You will have to verify you methods in light of these behaviours.

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

AlbertoAloe — Thu, 07 Dec 2017 18:46:47 GMT

Dan,

thanks for your reply.

Actually I took into account that when running cell statistics. Also when getting the mean from the mosaic I've been also carefull to mosaic properties that can prevent the use of all the rasters.

Basically the mosaic is a cube of 366 overlapping rasters. I just want the mosaic to return the mean of the 366 rasters......it works but when I make a percent variation out of cell stats versus mosaic layer I have an average of 3% and a maximum of 10%.

I'm attaching a simple test scenario with the 366 rasters (File Year2008TestRaster.zip) as well as what I get from running mean cell statistics (file Yr2008TestRasterMean.zip) and what I get from the mosaic (pointing to the same set of 366 rasters) with the operator mean (File Yr2008TestRasterMosMean.zip).

If you (or somebody else..) have few minutes to tell me how to get the same values from the mosaic.......

There must be something I misunderstood about mosaic datasets

Thanks a lot

Alberto

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Sun, 12 Dec 2021 01:46:28 GMT

Ok... lets have a look

in_ras0 = r"C:\Temp\.......\Yr2008TestRasterMosMean.tif"  # ---- mosaic mean
in_ras1 = r"C:\Temp\.......\Yr2008TestRasterMean.tif"     # ---- plain gnarly
r0 = arcpy.RasterToNumPyArray(in_ras0)  # ---- convert both to rasters
r1 = arcpy.RasterToNumPyArray(in_ras1)  # ---- nodata wasn't specified so I had 
r0m = np.ma.masked_where(r0<0.0, r0)    # to track it down and produce a 'mask'
r1m = np.ma.masked_where(r1<0.0, r1)    # for both

r0m.shape, r1m.shape      # ---- both the same shape
((60, 120), (60, 120))

r0m.size, r1m.size        # ---- total number of cells
(7200, 7200)

r0m.count(), r1m.count()  # ----- total number with values
(6274, 6274)

r0m.mean()                # ----- overall mean for r0m
29.678883088938477

r1m.mean()                # ----- ditto for r1m
30.755663253108064

r0m.max(), r1m.max()      # ----- hmmmmm
(1615.9902, 1670.0612)

r0m.min(), r1m.min()      # ------ hmmmmm
(0.00056408811, 0.00051229505)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

so the shapes, size and count of the two arrays/rasters are the same interms of the values given.

Which suggests that if the results were the result of processing then something is 'off' in the stack or the processing is different in the two methods.

The float32 dtype doesn't break anything.

I will let you ponder and suggest anything else for me to look at

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

AlbertoAloe — Thu, 07 Dec 2017 21:00:50 GMT

Dan,

Year2008TestRasterMean.tif (output from cell statistics) is OK because it's exactly what I get using other methods (Aggregation of the numpy array from the original netcdf, using sample in sa...).

The problem is in the mosaic dataset. It just references the same rasters....I cannot get it !!

Basically I set the mosaic as follows:

Mosaic dataset in a file geodatabase with ETRS_89_LAEA projection
"Add rasters", pointing to the folder where I have the 366 rasters I gave you , without building overviews
I then set the mosaic operator to Mean
I set "MaximumNumber of Rasters per Mosaic" = 400 to include them all. This property if not set correctly would have an impact on the mosaic operator limiting the number of rasters used

After I make a mosaic layer (dragging and dropping in ArcMap or using the gp tool) double checking that the default mosaic operator still is 'mean'. I run copy raster....and I get what you saw.

If you had time to try something I would ask you to make a mosaic dataset out of the 366 rasters and see if you can get the mosaic mean that matches the one from cell stats (or one that matches mine).

Thanks again

Alberto

P.S. If somebody else would be able to confirm my output from the mosaic dataset I would start to think that there is something weird in the mosaic algorithm

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Thu, 07 Dec 2017 21:39:21 GMT

So my numpy r1m (the masked array version of r2008TestRasterMean.tif was ok?

And you can confirm that they all share.

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Sun, 12 Dec 2021 01:46:31 GMT

I made a (366, y, x) array from your individual tiffs.

First question 366? just to be sure that each is a day... I have no intention of validating the inputs

Here is what I got

folder = r"C:\Temp\ ...... "
arcpy.env.workspace = folder             # ---- set the workspace
rasters = arcpy.ListRasters("*", 'tif')  # ---- read all the tiff's... hope they belong
a_s = []

for raster in rasters:
    a_s.append(arcpy.RasterToNumPyArray(raster))  # ---- convert to arrays and store

r = np.asarray(a_s)
r_m = np.ma.masked_where((r<0.0) | (r>10000), r)  # ---- masked using high/low guess

r_m.min()   #  0.0
r_m.max()   #  6128.4219
r_m.mean()  #  30.755659143207026

# ---- there appeared to be some that didn't have a nodata set or it was in error
# hence my dual masking‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Advise....

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

AlbertoAloe — Thu, 07 Dec 2017 22:23:15 GMT

Dan,

I'm out of office... I can only see at your figures but it completely makes sense for me. Your r0m.max() and r01m.max() are exactly the values at the location with max discharge (po river outlet)

In my case I did a per cell percent variation calculating a raster like this:

perc var = abs(Yr2008TestRasterMosMean - Yr2008TestRasterMean) / Yr2008TestRasterMean

Statistics on percvar are a mean of 3% and a max of 10%. You see at cell with max it should be 1670 but I get 1615. Why ?

Why Yr2008TestRasterMosMean is different from Yr2008TestRasterMean (which I know is correct ) ? What have I done wrong when obtaining the mean mosaic layer ?

I do not need the mosaic dataset to do these things but it's a convenient way to expose netcdf and multidimensional stuff to people that do not know how to use numpy, netcdf4 and so on...Besides, ESRI is advertizing it this way.

Thanks Dan

Alberto

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Thu, 07 Dec 2017 22:29:49 GMT

Maybe someone else will weigh in..

my only assumption was that 366 rasters needed to be included and that they shared all cell commonly.

The only thing else to do would be to examine each raster for nodata consistency.... in numpy I trust,

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

AlbertoAloe — Thu, 07 Dec 2017 22:35:06 GMT

Dan,

2008 was a leap year...366 are days.

Alberto

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Thu, 07 Dec 2017 22:43:42 GMT

ahhh! Ok that ruled out one thought I had

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

AlbertoAloe — Thu, 07 Dec 2017 22:48:56 GMT

Dan,

I think that if you made r_m.mean() on the temporal axis (axis = 0) and converting back to raster you should get Yr2008TestRasterMean. We know that this works. The problem is the mosaic dataset that is supposed to properly calculate the mean of a pile of rasters as an aggregation on the temporal dimension (or whatever we want to call it).

Thanks again for your help

Alberto

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Thu, 07 Dec 2017 23:14:33 GMT

Just finished... I got the same results as you did converting the mean along axis 0.

I will investigate 'where' the differences occur..

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Sun, 12 Dec 2021 01:46:34 GMT

Differences... ok... the mosaic mean behaves differently, they get pretty close when you move away from the edges (, but they are 'exact' in their behaviour of calculating the mean. I also can't see where the 'float32' dtype would matter. Someone should try the mosaic data set in pro IF 'float64' is the default...

# ---- top 5x5 block of values

r_mean.data[:5, :5]  # ---- tiffs to array, using numpy, mean on axis= 0
 
array([[ 10.94,   1.05,   1.2 ,   1.14,   0.94],
       [ 16.11,   2.31,   3.41,   2.19,   0.6 ],
       [  1.06,   4.59,   1.22,   1.99,   2.45],
       [  1.17,   5.82,   0.86,   0.56,   2.86],
       [  6.88,   0.93,   2.95,   0.42,   0.45]])

r1m.data[:5, :5]     # ---- Yr2008TestRasterMean.tif to array, masked 
 
array([[ 10.94,   1.05,   1.2 ,   1.14,   0.94],
       [ 16.11,   2.31,   3.41,   2.19,   0.6 ],
       [  1.06,   4.59,   1.22,   1.99,   2.45],
       [  1.17,   5.82,   0.86,   0.56,   2.86],
       [  6.88,   0.93,   2.95,   0.42,   0.45]], dtype=float32)

r0m.data[:5, :5]     # ---- Yr2008TestRasterMosMean.tif the mosaic

array([[ 11.09,   1.05,   1.2 ,   1.13,   0.93],
       [ 16.26,   2.31,   3.4 ,   2.17,   0.61],
       [  1.06,   4.57,   1.2 ,   1.97,   2.44],
       [  1.16,   5.78,   0.87,   0.56,   2.85],
       [  6.84,   0.93,   2.95,   0.42,   0.44]], dtype=float32)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Fri, 08 Dec 2017 18:07:15 GMT

Cody Benkelman‌ ... thoughts? or who to flag on this issue?

This doesn't apply since the extents are identical http://support.esri.com/en/bugs/nimbus/QlVHLTAwMDEwMTQ2MA==

There are tool many bugs for mosaic dataset, but nothing within the last year (as search on http://support.esri.com/en/)

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

AlbertoAloe — Fri, 08 Dec 2017 18:58:36 GMT

Dan,

thanks for taking care of this..

I'm still out of office (the 8th of December is bank holiday in Italy).

On Monday I can make other tests...

Out of the many I already did one was with ArcGIS Pro 2 getting (if I remember correctly) the same issues.

I agree that float32 cannot be an issue. You can try to use dtype 64 when averaging over axis = 0 and you would get irrelevant differences. Also ArcGIS Pro is 64 bit application but I think that if the rasters (and the mosaic dataset) are 32 bit nothing should change except the ability to reference more than 2 GB of memory (but this is not the case given the very small size of the test data).

I'll keep you posted

Thanks again

Alberto

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

AlbertoAloe — Tue, 12 Dec 2017 18:02:00 GMT

Dan,

still I cannot figure out what is going on....

I tried to check mosaic dataset properties in order to find the reason for observed difference among mean calculated with mosaic operator and cell statistics on source rasters

I'am attaching a python script with the geoprocessing chain I run in ArcGIS 10.5.1 for mosaic creation and setting. It can be synthesized as follows:

Create a mosaic dataset inside a file geodatabase
Load the mosaic pointing to the 366 rasters I sent you before (previous attachment)
Set the mosaic properties through gp function (mosaic method to none, operator to mean and other stuff.....)

Then, I input mosaic dataset to "copy raster" (or alternatively I can make a mosaic layer keeping the default properties) which should output a raster that SHOULD BE IDENTICAL to the one created with cell statistics. As we have seen this is not the case.

A note on ArcGIS Pro: I get the same exact raster I get in ArcGIS 10.5.1 when running copy raster against the mosaic dataset.

Another detail: I'm attaching the output from copy raster Yr2008TestRasterMosMeanV2.tif (based on these settings) that is slightly different from the one I sent you before (Yr2008TestRasterMosMean.tif). I found out that the reason for this is the mosaic method :

When set to None you would get Yr2008TestRasterMosMeanV2.tif. In this case the sorting of the rasters are based on the ObjectID
When set to "By Attribute" with StdTime field and base value '2008/01/01' you would get a raster that is identical to Yr2008TestRasterMosMean.tif

Given that:

For the sake of calculating our mean the sorting is irrelevant
I purposely scrambled ObjectID and StdTime in a way that sorting by ObjectID is different from sorting by StdTime

....there must be something wrong in the mosaic dataset

Alberto

P.S. I gotta solve this otherwise I cannot sleep !!

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Tue, 12 Dec 2017 18:21:10 GMT

I would report this since we both seem to replicate the same results from the standalone images using cellstatistics and numpy. don't know what is going on in the mosaic data set, but it has to be some 'setting' or another

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

AlbertoAloe — Tue, 12 Dec 2017 18:26:31 GMT

Thanks Dan.

Have a nice day

Alberto

Re: Mosaic dataset operator versus cell statistics - What is wrong ??

DanPatterson_Retired — Sun, 21 Jan 2018 20:21:56 GMT

Any news on this issue Alberto?