Rounding with with raster calculator

2839
4
Jump to solution
11-02-2016 03:50 PM
forestknutsen1
MVP Regular Contributor

I would like to round some elevation values to the nearest 100th in my a set of rasters. 

I would like to walk through the numbers for one cell.

1) raster_0 * 100 + 0.5

179.519989 * 100 + 0.5 = 17952.498047

2) Int(raster)

Int(17952.498047) = 17952

3) Float(raster)

Float(17952) = 17952.000000

4) raster / 100

17952.000000 / 100 = 179.520004

What is up with the 4 in the millionths place?

Raster properties.

0 Kudos
1 Solution

Accepted Solutions
DanPatterson_Retired
MVP Esteemed Contributor

17952.000000 / 100 = 17952.000004

is wrong to begin with... 

Glad you fixed it

>>> 17952.000000 / 100 #17952.000004
179.52
‍‍

... but even if it was, it is showing ... floating point representation  Floating point - Wikipedia a long but valuable read

don't worry about it unless you plan to compare for exact floating/double values, then you need to use a tolerance about your number.

There is nothing you can do about it really, other than cosmetic things or keep your values in long integer but just remember that when you do calculations.

View solution in original post

4 Replies
DanPatterson_Retired
MVP Esteemed Contributor

17952.000000 / 100 = 17952.000004

is wrong to begin with... 

Glad you fixed it

>>> 17952.000000 / 100 #17952.000004
179.52
‍‍

... but even if it was, it is showing ... floating point representation  Floating point - Wikipedia a long but valuable read

don't worry about it unless you plan to compare for exact floating/double values, then you need to use a tolerance about your number.

There is nothing you can do about it really, other than cosmetic things or keep your values in long integer but just remember that when you do calculations.

View solution in original post

forestknutsen1
MVP Regular Contributor
0 Kudos
DanPatterson_Retired
MVP Esteemed Contributor

Forest... nothing has changed in current versions of python 

15. Floating Point Arithmetic: Issues and Limitations — Python 3.6.0b4 documentation 

I can't remember if raster is now 32 bit or not.  If stuff is mission critical, numpy offers an extra layer of overconfidence.

>>> # just the numbers in float32.... no problem
>>>
>>> a = np.arange(3, dtype='float32')
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*a)
'0.000000000000000000 1.000000000000000000 2.000000000000000000'
>>>
>>> # now lets divide those float32 by 10.
>>>
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*a/10.)
'0.000000000000000000 0.100000001490116119 0.200000002980232239'
>>>
>>> # change the array of numbers to float64 and redo the division
>>>
>>> b = np.arange(3, dtype='float64')
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*b/10.)
'0.000000000000000000 0.100000000000000006 0.200000000000000011'
>>>
>>> #  not bad... but still not what most expect
0 Kudos
DanPatterson_Retired
MVP Esteemed Contributor

now that the answer is fixed, lets explore floating point representation in an environment where we can control its display 

a = np.array(17952.0, dtype='float16')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float16')
>>> a = np.array(17952.0, dtype='float32')
>>> a/100
179.52000000000001
>>> a = np.array(17952.0, dtype='float64')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float64')
>>> a = np.array(17952.0, dtype='float128')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float64')
>>>
>>> # so all the above show the strange behaviour until ...
>>>
>>> a= 17952.0/100
>>> a
179.52
>>>
>>> # now what is really lurking behind, but isn't show, since normally only 6 decimals
>>> # are shown unless forced otherwise
>>>
>>> "{:<16.12f}".format(a/100.)
'1.795200000000  '
>>> "{:<20.16f}".format(a/100.0)
'1.7952000000000001  '

Thats why I said don't worry about it... the value you get will depend on the machine architecture you have as well.