# Rounding with with raster calculator

4630
4
Jump to solution
11-02-2016 03:50 PM
MVP Regular Contributor

I would like to round some elevation values to the nearest 100th in my a set of rasters.

I would like to walk through the numbers for one cell.

1) raster_0 * 100 + 0.5

179.519989 * 100 + 0.5 = 17952.498047

2) Int(raster)

Int(17952.498047) = 17952

3) Float(raster)

Float(17952) = 17952.000000

4) raster / 100

17952.000000 / 100 = 179.520004

What is up with the 4 in the millionths place?

Raster properties.

1 Solution

Accepted Solutions
MVP Legendary Contributor

17952.000000 / 100 = 17952.000004

is wrong to begin with...

Glad you fixed it

``````>>> 17952.000000 / 100 #17952.000004
179.52
‍‍‍‍‍``````

... but even if it was, it is showing ... floating point representation  Floating point - Wikipedia a long but valuable read

don't worry about it unless you plan to compare for exact floating/double values, then you need to use a tolerance about your number.

There is nothing you can do about it really, other than cosmetic things or keep your values in long integer but just remember that when you do calculations.

4 Replies
MVP Legendary Contributor

17952.000000 / 100 = 17952.000004

is wrong to begin with...

Glad you fixed it

``````>>> 17952.000000 / 100 #17952.000004
179.52
‍‍‍‍‍``````

... but even if it was, it is showing ... floating point representation  Floating point - Wikipedia a long but valuable read

don't worry about it unless you plan to compare for exact floating/double values, then you need to use a tolerance about your number.

There is nothing you can do about it really, other than cosmetic things or keep your values in long integer but just remember that when you do calculations.

MVP Regular Contributor

ooops.... I fixed my typo.

Thanks for the info Dan.

MVP Legendary Contributor

Forest... nothing has changed in current versions of python

I can't remember if raster is now 32 bit or not.  If stuff is mission critical, numpy offers an extra layer of overconfidence.

``````>>> # just the numbers in float32.... no problem
>>>
>>> a = np.arange(3, dtype='float32')
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*a)
'0.000000000000000000 1.000000000000000000 2.000000000000000000'
>>>
>>> # now lets divide those float32 by 10.
>>>
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*a/10.)
'0.000000000000000000 0.100000001490116119 0.200000002980232239'
>>>
>>> # change the array of numbers to float64 and redo the division
>>>
>>> b = np.arange(3, dtype='float64')
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*b/10.)
'0.000000000000000000 0.100000000000000006 0.200000000000000011'
>>>
>>> #  not bad... but still not what most expect‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍``````
MVP Legendary Contributor

now that the answer is fixed, lets explore floating point representation in an environment where we can control its display

``````a = np.array(17952.0, dtype='float16')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float16')
>>> a = np.array(17952.0, dtype='float32')
>>> a/100
179.52000000000001
>>> a = np.array(17952.0, dtype='float64')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float64')
>>> a = np.array(17952.0, dtype='float128')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float64')
>>>
>>> # so all the above show the strange behaviour until ...
>>>
>>> a= 17952.0/100
>>> a
179.52
>>>
>>> # now what is really lurking behind, but isn't show, since normally only 6 decimals
>>> # are shown unless forced otherwise
>>>
>>> "{:<16.12f}".format(a/100.)
'1.795200000000  '
>>> "{:<20.16f}".format(a/100.0)
'1.7952000000000001  '
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍``````

Thats why I said don't worry about it... the value you get will depend on the machine architecture you have as well.