topic Re: Excluding "NoData" in raster percentile calculation in Python Questions

Excluding "NoData" in raster percentile calculation

anTonialcaraz — Sat, 11 Dec 2021 22:45:58 GMT

Hi,

I'm trying to calculate percentiles out of a floating point raster.

It works BUT it includes the NoData in the calculation so I don't get what I need.

Is there a way to exclude the NoData from this calculation?

Many thanks

env.workspace = r"D:\PROGRAMMES\LFP_Source_Rocks\ArcGIS\00_LFP_GLOBAL\00_UPWELLING\OUTPUT_Test.gdb"
raster = env.workspace + "\\" + "percentile"
array = arcpy.RasterToNumPyArray(raster)
array = scipy.sort(array)
per = 10
a = scipy.percentile(array,per)
print a

Re: Excluding "NoData" in raster percentile calculation

ToddBlanchette — Thu, 16 Jul 2015 14:41:24 GMT

Hi Toni,

The arcpy.RasterToNumPyArray tool has a parameter called 'nodata_to_value'. This parameter allows you to set a value to assign to your input raster's nodata. The default value is "None," which is probably what's messing up your calculations.

Tool help - ArcGIS Help (10.2, 10.2.1, and 10.2.2)

Cheers,

Todd

Re: Excluding "NoData" in raster percentile calculation

anTonialcaraz — Thu, 16 Jul 2015 14:52:46 GMT

Hi Todd,

Many thanks for your reply.

I'm aware NoData can be assigned to a particular value. But, wouldn't that then "corrupt" the percentile calculation? as in this case there would be a lot of say value 1 (if for example I assign NoData to 1).

I'm working with global rasters with values present only 250km from the coast.

I think for the percentile calculation to be correct only the real list of values should be taken into account excluding the NoData values.

Or maybe I'm not quite right...

Cheers

Re: Excluding "NoData" in raster percentile calculation

DanPatterson_Retired — Thu, 16 Jul 2015 15:07:17 GMT

check your duplicate thread

Re: Excluding "NoData" in raster percentile calculation

ToddBlanchette — Thu, 16 Jul 2015 15:15:12 GMT

Sorry Toni, you are correct in that you still have to deal with the noData in some way as to not taint your calculations - I thought you were just wondering if you could modify the noData.

You could probably use an array mask on your numpy array to exclude the values you don't want to include.

More info on how to do this here The numpy.ma module — NumPy v1.9 Manual

Cheers,

Todd

Re: Excluding "NoData" in raster percentile calculation

anTonialcaraz — Sat, 11 Dec 2021 22:46:01 GMT

Thanks Todd/Dan,

I'm trying to use numpy mask module. Not sure I'm getting the right syntax when calling the "NoData" (Null, NoData...) values.

I don't quite see how to do so in the documentation. This is how my code looks at the moment.

Any further help would be greatly appreciate it.

env.workspace = r"D:\PROGRAMMES\LFP_Source_Rocks\ArcGIS\00_LFP_GLOBAL\00_UPWELLING\OUTPUT_Test.gdb"

raster = env.workspace + "\\" + "percentile"
array = arcpy.RasterToNumPyArray(raster)
marray = numpy.ma.masked_values(array,NULL)
sarray = scipy.sort(marray)
per = 20
a = scipy.percentile(sarray,per)
print a

Re: Excluding "NoData" in raster percentile calculation

anTonialcaraz — Fri, 17 Jul 2015 10:53:21 GMT

Hi again,

I've been trying also to set NoData to a value and then use "ma.masked_values" to exclude that value from the percentile calculation but it looks as if the value is not excluded.

Re: Excluding "NoData" in raster percentile calculation

anTonialcaraz — Sat, 11 Dec 2021 22:46:04 GMT

Looks as if I finally managed to get it running. Code looks like this:

env.workspace = r"D:\PROGRAMMES\LFP_Source_Rocks\ArcGIS\00_LFP_GLOBAL\00_UPWELLING\OUTPUT_Test.gdb"
raster = env.workspace + "\\" + "percentile"
array = arcpy.RasterToNumPyArray(raster,nodata_to_value=10000000)
marray = numpy.ma.masked_values(array,10000000)
sarray = scipy.sort(marray)
per = 40
percen = numpy.percentile(sarray.compressed(),(per))
print percen

Thanks for your comments/help