topic Re: Rounding with with raster calculator in Python Questions

Rounding with with raster calculator

forestknutsen1 — Wed, 02 Nov 2016 22:50:35 GMT

I would like to round some elevation values to the nearest 100th in my a set of rasters.

I would like to walk through the numbers for one cell.

1) raster_0 * 100 + 0.5

179.519989 * 100 + 0.5 = 17952.498047

2) Int(raster)

Int(17952.498047) = 17952

3) Float(raster)

Float(17952) = 17952.000000

4) raster / 100

17952.000000 / 100 = 179.520004

What is up with the 4 in the millionths place?

Raster properties.

Re: Rounding with with raster calculator

DanPatterson_Retired — Fri, 10 Dec 2021 21:04:31 GMT

17952.000000 / 100 = 17952.000004

is wrong to begin with...

Glad you fixed it

>>> 17952.000000 / 100 #17952.000004
179.52
‍‍‍‍‍

... but even if it was, it is showing ... floating point representation Floating point - Wikipedia a long but valuable read

don't worry about it unless you plan to compare for exact floating/double values, then you need to use a tolerance about your number.

There is nothing you can do about it really, other than cosmetic things or keep your values in long integer but just remember that when you do calculations.

Re: Rounding with with raster calculator

DanPatterson_Retired — Fri, 10 Dec 2021 21:04:34 GMT

now that the answer is fixed, lets explore floating point representation in an environment where we can control its display

a = np.array(17952.0, dtype='float16')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float16')
>>> a = np.array(17952.0, dtype='float32')
>>> a/100
179.52000000000001
>>> a = np.array(17952.0, dtype='float64')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float64')
>>> a = np.array(17952.0, dtype='float128')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float64')
>>>
>>> # so all the above show the strange behaviour until ...
>>>
>>> a= 17952.0/100
>>> a
179.52
>>>
>>> # now what is really lurking behind, but isn't show, since normally only 6 decimals
>>> # are shown unless forced otherwise
>>>
>>> "{:<16.12f}".format(a/100.)
'1.795200000000  '
>>> "{:<20.16f}".format(a/100.0)
'1.7952000000000001  '
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Thats why I said don't worry about it... the value you get will depend on the machine architecture you have as well.

Re: Rounding with with raster calculator

forestknutsen1 — Thu, 03 Nov 2016 02:29:41 GMT

ooops.... I fixed my typo.

Thanks for the info Dan.

14. Floating Point Arithmetic: Issues and Limitations — Python 2.7.12 documentation

Re: Rounding with with raster calculator

DanPatterson_Retired — Fri, 10 Dec 2021 21:04:37 GMT

Forest... nothing has changed in current versions of python

15. Floating Point Arithmetic: Issues and Limitations — Python 3.6.0b4 documentation

I can't remember if raster is now 32 bit or not. If stuff is mission critical, numpy offers an extra layer of overconfidence.

>>> # just the numbers in float32.... no problem
>>>
>>> a = np.arange(3, dtype='float32')
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*a)
'0.000000000000000000 1.000000000000000000 2.000000000000000000'
>>>
>>> # now lets divide those float32 by 10.
>>>
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*a/10.)
'0.000000000000000000 0.100000001490116119 0.200000002980232239'
>>>
>>> # change the array of numbers to float64 and redo the division
>>>
>>> b = np.arange(3, dtype='float64')
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*b/10.)
'0.000000000000000000 0.100000000000000006 0.200000000000000011'
>>>
>>> #  not bad... but still not what most expect‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍