topic Re: How to normalize feature data between 0-1 in ArcGIS Pro? in Spatial Statistics Questions

How to normalize feature data between 0-1 in ArcGIS Pro?

HuangleiPan1 — Mon, 11 Mar 2019 03:25:50 GMT

Hi,

I have a feature table, and want to normalize the data in a field between 0-1, and I know the normalization method should be "(x-min(x))/(max(x)-min(x) ". I have tried "add field---field calculator", but I have no idea how to use the functions. Could anyone help me to achieve it in ArcGIS Pro?

Thanks

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

EvgenyPanchenko1 — Mon, 11 Mar 2019 12:31:45 GMT

Hello Pan,

you'd need to use the Raster Calculator tool to employ this formula.

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

HuangleiPan1 — Mon, 18 Mar 2019 03:52:34 GMT

Thank you for you kind reply, Evgency. However, I have tens of fields to be normalized. If I convert all of them to rasters and then use the Raster Calculator Tool, it would be tedious. Could there be other methods to excute directly on feature dataset?

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

DanPatterson_Retired — Mon, 18 Mar 2019 04:13:12 GMT

If the fields are numeric... and you have a very large number, then I would suggest you export the table to a spreadsheet, normalize the data there, then rejoin the data back to the original file. You will have to make sure that after you normalize there, that you copy the results with non-duplicate field names... and no formula etc to a fresh clean spreadsheet then use the Excel To Table tool to bring it back into pro.

There is no 'batch' normalize function in Pro and scripting would take you far longer than copying and pasting a formula in excel to do the work.

My preference would be to do the normalization using numpy and python, but I suspect that isn't an easier option for you.

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

DanPatterson_Retired — Mon, 18 Mar 2019 08:05:22 GMT

To atone for recommending a spreadsheet.... I have provided a quick demonstration using numpy and the builtin arcpy.da module TableToNumPyArray and NumPyArrayToTable functionality... which I wish was used and advertised more. It is lonely in the numpy/arcpy world... must develop a web widget for this stuff

/blogs/dan_patterson/2019/03/18/math-and-stats-with-numpy-normalize-data

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

HuangleiPan1 — Tue, 19 Mar 2019 03:23:21 GMT

Thank you very much, Dan! After reading your script, I have solved the problem !

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

DanPatterson_Retired — Tue, 19 Mar 2019 03:50:21 GMT

Glad it worked..

I updated the blog to add examples if you needed to normalize data by column (as you need), but also by row or overall, depending on how your data are aranged or what it represents. Keep it in mind. It would be a bit beyond the field calculator, but arcpy and numpy interplay quite nicely to solve problems

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

curtvprice — Tue, 19 Mar 2019 05:48:10 GMT

A more traditional approach (still a lot less work IMHO, sorry Dan):

1. Find the min and max values, let's say they are 10 and 900. (right click field in table view, Statistics)

2. Add a field

3. Run Calculate field with this expression (VFLD is your field that ranges from 0 to 900) (Use Python parser)

min + (max-min) * (value - 10.0) / (max-min)

10 + ((900 - 10) * (!VFLD! - 10.0) / (900 - 10))‍

I should mention that what you are suggesting is not "normalizing" the data but instead doing a min-max "stretch" of the data. To Normalize the data you would convert the values to Z scores using the mean and standard deviation (replace mean and std with the numbers you get from the Statistics run):

(!VFLD! - mean) / std‍

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

DanPatterson_Retired — Tue, 19 Mar 2019 09:30:20 GMT

Curtis the statistical and image image stretch are both forms of Normalization - Wikipedia

I prefer to think of it as the process by which "a basic standard by which value is computed" (Numéraire like currency) since an upper range is not necessarily required but a 'base' of comparison is required

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

HuangleiPan1 — Tue, 19 Mar 2019 13:30:38 GMT

Dan, I find a field of my data couldn't been processed by the script, the results for it were all Null. However, the datatype(double) and other properties of this field are same as the others . The only difference is that the data of this field were calculated from some other fields using Field Calculator Tool. The warning message is below:

Warning (from warnings module):
File "C:\Users\pan\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone2\Lib\site-packages\numpy\core\fromnumeric.py", line 83
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
RuntimeWarning: invalid value encountered in reduce

I only know some basics of Python, so I can not figure out the problem here by myself.

Do you know why the error came?

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

DanPatterson_Retired — Sun, 12 Dec 2021 04:53:49 GMT

def normalize(a, axis=None):
    # a is a (n x dimension) np.array
    tmp = a - np.nanmin(a, axis=axis)
    out = tmp / (np.nanmax(tmp, axis=axis) - np.nanmin(tmp, axis=axis))
    return out‍‍‍‍‍

You didn't indicate that there was the possibility of nulls in the field... that can be fixed using 'nan' functions, but ptp doesn't have an equivalent so the function would have to be

In this fashion, <null> aka nodata, None values are omitted from the calculation.

Re: How to normalize feature data between 0-1 in ArcGIS Pro?

HuangleiPan1 — Wed, 20 Mar 2019 07:31:19 GMT

Thank you , the new function worked well on this field!