topic Re: How do I normalize raster data? in Data Management Questions

How do I normalize raster data?

TaylerHamilton — Thu, 26 Jan 2012 21:30:03 GMT

I want to combine different raster layers, but need the data to be measured at the same numerical scale. Therefore, I would like to normalize all of the different layers so I can add them together, and then renormalize them.

I am not very good with statistics in general, and am unsure of how to even normalize a data set. I know there are multiple equations to do this, but don't know which one is right for me.

But if we were to use the equation:

Z = X - u / std

Where, Z - the normalized value;
u - mean; and,
std - standard deviation;

How would I calculate this in a raster?

I have tried using "Calculate Statistics" (Data Management Tools) because I assumed it would calculate mean and standard deviation for me, but I don't know where the output goes, as I can't choose that in the window. Also, the information on what this tool actually calculates is pretty limited in desktop help.

I have thought about using Focal Statistics to calculate mean and standard deviation individually, but I run into the problem of what scale to use - I want it to include the entire raster in the calculation. This tool would give me different outputs for mean and standard deviation, which I could then put into the raster calculator to solve the normalization equation.

Can someone please let me know if using Focal Statistics is the best way to go about this? Am I even using the right equation to normalize my data?

Any input is GREATLY appreciated!!

Re: How do I normalize raster data?

JeffreyEvans — Fri, 27 Jan 2012 13:14:12 GMT

I have a toolbox available that has a normalize options in the Statistics > transformations tool.

http://conserveonline.org/workspaces/emt/documents/arcgis-geomorphometrics-toolbox/view.html

Re: How do I normalize raster data?

TaylerHamilton — Fri, 27 Jan 2012 20:25:52 GMT

Thanks! I will give it a shot and let you know if its what I am looking for!

Re: How do I normalize raster data?

WendyProudfoot — Fri, 16 Mar 2012 17:13:06 GMT

I have a toolbox available that has a normalize options in the Statistics > transformations tool.

http://conserveonline.org/workspaces/emt/documents/arcgis-geomorphometrics-toolbox/view.html

Hi Jeffrey,

This tool sounds great.

Is the tool only available for v10? Unfortunately I'm still on v9.3.1 and won't be moved to v10 for a while.

Wendy

Re: How do I normalize raster data?

JeffreyEvans — Tue, 20 Mar 2012 16:04:03 GMT

Sorry, I should have been clear that the toolbox was only compatible with ArcGIS 10. You are on the right track with calculate statistics. Make sure you set the skip value to 1, then add the raster to ArcMap (or navigate to it in ArcCatalog) and right click on it and select properties. In the layer properties box select the source tab and scroll down to the bottom and you will see the statistics section and can retrieve the min, max, mean and stdv (can copy and paste the values). You can then use the values to perform a normalization on your raster in the raster calculator (Arctoolbox > Spatial Analyst Tools > Map Algebra > Raster Calculator) with the following syntax.

(x - mean) / stdv
where; x is your raster, mean and stdv are the real values of the respective statistical moments.

The above formula from your original post does not transform to a standard variable space. This transformation is intended to scale the mean to 0 and stdv to 1 while maintaining the shape of the original distribution. The resulting data range is dictated by the range in your original data. This formula, in effect, makes the negative and positive bounds symmetrical (e.g., -1 to 1). If the original data distribution is non-normal the results can be unexpected. If you just want your data in the same scale and it is all positive, you could just perform a "row standardization" by dividing your raster by its max value to (mostly) return a range of 0-1. Here is a method that accounts for negative values and also reliably returns a range of 0-1. ( x - min(x) ) / ( max(x) - min(x) )

If you have access to R, the behavior of these three transformations can be readily observed given a normal and skewed distribution. Note that you can plot the distribution of any of the resulting transformations and it does not change shape, just range. Here is the code (just copy and paste).

############################
# Based on a non-normal distribution
############################
x <-runif(100,1,100)
summary(x)
plot(density(x))

x1 <- ( x - mean(x) ) / sd(x)
summary(x1)

x2 <- x / max(x)
summary(x2)

x3 <- ( x - min(x) ) / ( max(x) - min(x) )
summary(x3)

# When distribution has negative values (note that the regular row standardization [x/max(x)] does not scale correctly)
x[1] <- -10
x2 <- x / max(x)
x3 <- ( x - min(x) ) / ( max(x) - min(x) )
summary(x2);summary(x3)

############################
# Based on a non-normal distribution
############################
x <- rweibull(1e5,1.5,33)
summary(x)
plot(density(x))

x1 <- ( x - mean(x)) / sd(x)
summary(x1)

x2 <- x / max(x)
summary(x2)

x3 <- ( x - min(x) ) / ( max(x) - min(x) )
summary(x3)

Re: How do I normalize raster data?

Ching-AnChiu — Fri, 04 Jan 2013 08:25:49 GMT

Dear all,
Geomorphometry and Gradient Metrics Toolbox (GGMT) is a powerful tool.
Thanks for Jeffrey�??s works!

For to normalize raster data, I tried three ways:
1) Using raster calculator in ArcGIS, (x - mean) / stdv --> ( x - min(x) ) / ( max(x) - min(x) )
2) Using GGMT, Statistics --> transformations --> normalize
3) Using GGMT, Statistics --> transformations --> standardize --> stretch 0~1
The new layers transformed through the three different ways are the same results.

My question is following:
Before running Species Distribution Models such as MaxEnt, I don't know whether to normalize, standardize, and rescale (0~1) the environmental raster layers is a necessary step.
If that is necessary!
How should I do that in GGMT?

Thank you in advance for your assistance!

Re: How do I normalize raster data?

JeffreyEvans — Mon, 07 Jan 2013 16:55:54 GMT

The decision to transform your data is dependent on the type of model that you are using and the results of an exploratory analysis of your data. If you were using a parametric method such as OLS (Ordinary Least Square Regression) or ENFA (Ecological Niche Factor Analysis), I would certainly transform my covariates. However, the most powerful models for species distribution modeling (MaxEnt, Random Forests, etc...) are nonparametric and, as such, do not have IID or distributional assumptions. Because of this no data transformations are necessary.

Re: How do I normalize raster data?

Ching-AnChiu — Wed, 09 Jan 2013 13:04:20 GMT

Hi Jeffrey,

Thanks for the reply!

When I want to compare different methods (including MaxEnt and ENFA)
whether it is a good idea to calculate raster
using (x - mean) / stdv --> (x - min(x)) / (max(x) - min(x))
for all environmental variables before running species distribution models.

What is the difference between �??normalize�?� and �??standardize�?� in Geomorphometry and Gradient Metrics Toolbox.
Their algorithms are�?�

Re: How do I normalize raster data?

JenHooper — Wed, 20 Mar 2013 17:24:54 GMT

Once again,
thanks superman jeff evans!
jen hooper

Re: How do I normalize raster data?

VitorVasconcelos — Tue, 03 Dec 2013 20:39:58 GMT

Hello, friend.

Remember that, for standardizing data, you assume that this data is in a normal distribution. If the data is not in a normal distribution, it is advisable to use percentile rank instead of normalization. A good explanation about the theory and about how to do it in SPSS are in these links:

http://www.psychstat.missouristate.edu/introbook/sbk14m.htm

http://blogs.perficient.com/businessintelligence/2012/03/21/ranking-your-cases-ibm-spss-statistics/

Anyway, if you are using ArcGIS, a good option is to stretch your raster using "Equalize Histogram", than export the raster ("Data" -> "Export data") using the option "use renderer" and setting the "No Data Value" to 0. This will give you a percentile rank from 1 to 255 and you can re-scale it dividing your raster values by 2.5.

Have fun!

Re: How do I normalize raster data?

DanielAmrine — Wed, 23 Jul 2014 14:50:42 GMT

I am in a similar situation but I would like to reclassify my raster data into a correlation range of -1 to 1. I know I can classify my original raster and use the re-class tool, but I would prefer to use the raster calculator to do this so I can preserve the "Shape" of my raster.

I tried the (x-min(x))/(max(x)-min(x)) and it normalized to 0-1.

Thanks for your help!

Dan

Re: How do I normalize raster data?

FlorianHoedt2 — Wed, 23 Jul 2014 15:51:49 GMT

You could use the ndvi as an example. Its arithmetics are explained in the arcgis help or the www. Sorry my mobile i low on battery and i can not provide links currently.

what gives this?;

( max(x) - min(x) ) / ( max(x) + min(x) ) // this is stupid sorry

this is not tested though.

beware the spelling >> send from my mobile

Re: How do I normalize raster data?

FlorianHoedt2 — Wed, 23 Jul 2014 16:14:21 GMT

i have done some test in excel to get it working:

( x - min(x) ) / (max(x) - min(x) ) -- your code to create 0 - 1 normalized raster --> let us call it raster1

( raster1 + 1) * -1 -- reverse the values 1-> 0 ; 0-> -1 ; call it raster2

( raster1 + raster2) -- -1 to 1

i am not a math or statistics guy though!

Re: How do I normalize raster data?

JeffreyEvans — Wed, 23 Jul 2014 17:24:48 GMT

Are you sure that you want to do this? You are, in effect, creating a two-tailed distribution where the negative tail is indicating a proportional negative response equal to the positive. If this is what you are after then you need to identify a "hinge point" that defines the inflection point, in the original distribution, indicating where the distribution will be centered on 0. This will provide relevant information on where the distribution changes to a negative influence. If the distribution is not centered then the result will be arbitrary. The equations for normalizing are not applicable here.

This will not necessarily provide a bounded -1 to 1 range but the equation for standardizing a distribution to a standard deviation of 1 and a mean of 0 thus, providing a Z-score, is: [(x - mean(x)) / standard deviation(x)]. You can play with the math to center on a different value. There is a good reason that we normally scale distributions to a 0-1 range.

The problem here is that a Z-score standardization assumes a Gaussian distribution, which you likely do not have. Overall I do not believe that this is a good idea. Perhaps if you provided some context I will be able to provide a relevant alternative.

If you data represents a known fixed range (e.g., correlations coefficients) and for some reason exhibit erroneous values, I would recommend just truncating them using a con statement (eg., con(raster < -1, -1, raster) ).

Re: How do I normalize raster data?

JeffreyEvans — Wed, 23 Jul 2014 17:28:26 GMT

NDVI is a ratio between two bands. The equation to calculate the ratio on a single vector will result in all zero values. The equation you provide is a ratio in the upper and lower tails of the distribution and will result in a single value.

Re: How do I normalize raster data?

DanielAmrine — Wed, 23 Jul 2014 18:08:00 GMT

My challenge is correlating magnetic (total magnetic intensity, rated to pole, and several passes),gravity (bouguer anomaly, several passes) and depth to oil and gas production data.

I used a "Statistics for Dummy" book to find the correlation equation and then used a combination of excel and arcmap to map and correlate values for 652 wells.

The problem i'm having is the correlation coefficient is supposed to fall within -1.0 to 1.0 but each grid ranges from -8.5 to 8.5 or so, i can see the trend but I want it to fit within -1 to 1.

i hope that provides some good context!

Thanks!

Dan

Re: How do I normalize raster data?

JeffreyEvans — Wed, 23 Jul 2014 19:18:17 GMT

Well, the problem sounds like your correlation approach is not working and forcing the range will not correct the problem. Also the type of correlation (Person, Kendall, Spearman) is important given the data. I would revisit your methodology to figure out why your correlation coefficients are off rather than trying to fix the issue post hoc. Any approach for deriving correlation coefficients at the raster cell level using a combination of ArcMAP and Excel are opaque at best and some details would be required to evaluate the approach.

I am not clear on why you want these correlations as a raster. If you have well locations, with associated depth, you could just assign the gravitational anomaly raster values using the "Extract Multi Values to Points" tool, export the resulting attribute table to a flatfile (eg., csv) and calculate the correlation coefficient in Excel (not that Excel is good for statistical analysis).

It is interesting that you mention this particular correlative relationship. I just published a paper in PLoS One (Evans & Kiesecker 2014) using gravitational anomaly data, along with other covariates, in a probabilistic model of non-conventional oil/gas. I used a nonparametric model and found that this particular relationship is highly nonlinear in nature. Because of this, any correlations are likely to be erroneous. I would imagine that it is time to move your analysis into a statistical software and it may also be a good point to consult a statistician.

Evans, J.S., J.M. Kiesecker (2014) Shale Gas, Wind and Water: Assessing the Potential Cumulative Impacts of Energy Development on Ecosystem Services within the Marcellus Play. PLoS ONE 9(2): e89210. doi:10.1371/journal.pone.0089210

If you were to produce a surface of depth, using Kriging or any such interpolation method, you could easily calculate a moving window correlation surface in R. Here is an example, that includes a Kriging estimate, for calculating a moving window correlation surface.

# Moving window correlation function

# x raster of x

# y raster of y

# dist distance (radius) of correlation window, the default AUTO calculates a window size

# ... additional arguments passed to the cor function

mwcor <- function(x, y, dist="AUTO", ...) {

require(sp)

require(spdep)

if ( (dist == "AUTO") == TRUE){

cs <- x@grid@cellsize[1]

dist = sqrt(2*((cs*3)^2))

} else {

if (!is.numeric (dist)) stop("DISTANCE MUST BE NUMERIC")

}

nb <- dnearneigh(coordinates(x),0, dist)

v=sapply(nb, function(i) cor(x@data[i,], y@data[i,], ...))

if( (class(v)=="numeric") == TRUE) {

v = as.data.frame(v)

} else {

v = as.data.frame(t(v))

}

coordinates(v) = coordinates(x)

gridded(v) = TRUE

( v )

}

# Example

require(gstat)
require(sp)
require(spdep)

data(meuse)
data(meuse.grid)
coordinates(meuse) <- ~x + y
coordinates(meuse.grid) <- ~x + y

# GRID-1 log(copper):
v1 <- variogram(log(copper) ~ 1, meuse)
x1 <- fit.variogram(v1, vgm(1, "Sph", 800, 1))

G1 <- krige(zinc ~ 1, meuse, meuse.grid, x1, nmax = 30)

gridded(G1) <- TRUE

G1@data = as.data.frame(G1@data[,-2])

# GRID-2 log(lead):
v2 <- variogram(log(lead) ~ 1, meuse)
x2 <- fit.variogram(v2, vgm(.1, "Sph", 1000, .6))

G2 <- krige(zinc ~ 1, meuse, meuse.grid, x2, nmax = 30)

gridded(G2) <- TRUE

G2@data <- as.data.frame(G2@data[,-2])

# Moving window correlation surface

gcorr <- mwcor(G1, G2, 500, method="spearman")

# Plot results

colr=colorRampPalette(c("blue", "yellow", "red"))

spplot(gcorr, col.regions=colr(100))

Re: How do I normalize raster data?

DanielAmrine — Wed, 23 Jul 2014 19:44:35 GMT

Jeffery,

I just downloaded your paper and i'm looking forward to reading it!

I've spent the last six years working as a mapping and geoscience technician in the Marcellus field. However my company had to go through some cuts and now i'm working on projects all over the country in terms of exploration.

The difficulty comes in correlating the data on a one to one relationship rather than subtracting the means from X and Y, multiplying them together, summing them and then dividing by the product of the X and Y standard deviations. Then dividing the result by the total number of values and subtracting by 1. That is the core of the process I used I just didn't Sum them since there where only 2 values (an X and Y) for each well.

For the grids I tried leaving out the sum but obviously it didn't work. My goal is to derive a correlation coefficient for each well in terms of the correlated data sets.

My main goal is to see if there are any spatial relationships to the correlation of these different values. Essentially we want to find out what kind of data will determine the success of an oil and gas reservoir by using any means possible!

I also tried copying and pasting your code into R and, I'm such a noob, I have no idea on how to fix the code so it will run. I will work on that!

Thank you for taking the time to reply!

Re: How do I normalize raster data?

WaseemAli — Sun, 02 Nov 2014 14:42:55 GMT

I applied the same equation in raster calculator of arcgis but give me the error while running. I thing there are some abnormal values in it. How can i remove such abnormalities first and then i would apply this formula to derive the ndvi.
Also comment about the formula NDVI - NDVImin / NDVImax - NDVImax called Green vegetation fraction (GVF). Hu and Jia 2010 used this same equation in his study to derive the green vegetation fraction and gives the ranges 0.20 and 0.70 as minimum and maximum respectively that have been taken from AVHRR data for the same region http://onlinelibrary.wiley.com/enhanced/doi/10.1002/joc.1984 but how can i use these values for the area where AVHRR data is not available as in the case of mine.