I would like to round some elevation values to the nearest 100th in my a set of rasters.
I would like to walk through the numbers for one cell.
1) raster_0 * 100 + 0.5
179.519989 * 100 + 0.5 = 17952.498047
2) Int(raster)
Int(17952.498047) = 17952
3) Float(raster)
Float(17952) = 17952.000000
4) raster / 100
17952.000000 / 100 = 179.520004
What is up with the 4 in the millionths place?
Raster properties.
Solved! Go to Solution.
17952.000000 / 100 = 17952.000004
is wrong to begin with...
Glad you fixed it
>>> 17952.000000 / 100 #17952.000004
179.52
... but even if it was, it is showing ... floating point representation Floating point - Wikipedia a long but valuable read
don't worry about it unless you plan to compare for exact floating/double values, then you need to use a tolerance about your number.
There is nothing you can do about it really, other than cosmetic things or keep your values in long integer but just remember that when you do calculations.
17952.000000 / 100 = 17952.000004
is wrong to begin with...
Glad you fixed it
>>> 17952.000000 / 100 #17952.000004
179.52
... but even if it was, it is showing ... floating point representation Floating point - Wikipedia a long but valuable read
don't worry about it unless you plan to compare for exact floating/double values, then you need to use a tolerance about your number.
There is nothing you can do about it really, other than cosmetic things or keep your values in long integer but just remember that when you do calculations.
ooops.... I fixed my typo.
Thanks for the info Dan.
14. Floating Point Arithmetic: Issues and Limitations — Python 2.7.12 documentation
Forest... nothing has changed in current versions of python
15. Floating Point Arithmetic: Issues and Limitations — Python 3.6.0b4 documentation
I can't remember if raster is now 32 bit or not. If stuff is mission critical, numpy offers an extra layer of overconfidence.
>>> # just the numbers in float32.... no problem
>>>
>>> a = np.arange(3, dtype='float32')
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*a)
'0.000000000000000000 1.000000000000000000 2.000000000000000000'
>>>
>>> # now lets divide those float32 by 10.
>>>
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*a/10.)
'0.000000000000000000 0.100000001490116119 0.200000002980232239'
>>>
>>> # change the array of numbers to float64 and redo the division
>>>
>>> b = np.arange(3, dtype='float64')
>>> "{:<20.18f} {:<20.18f} {:<20.18f}".format(*b/10.)
'0.000000000000000000 0.100000000000000006 0.200000000000000011'
>>>
>>> # not bad... but still not what most expect
now that the answer is fixed, lets explore floating point representation in an environment where we can control its display
a = np.array(17952.0, dtype='float16')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float16')
>>> a = np.array(17952.0, dtype='float32')
>>> a/100
179.52000000000001
>>> a = np.array(17952.0, dtype='float64')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float64')
>>> a = np.array(17952.0, dtype='float128')
>>> a/100
179.52000000000001
>>> a.dtype
dtype('float64')
>>>
>>> # so all the above show the strange behaviour until ...
>>>
>>> a= 17952.0/100
>>> a
179.52
>>>
>>> # now what is really lurking behind, but isn't show, since normally only 6 decimals
>>> # are shown unless forced otherwise
>>>
>>> "{:<16.12f}".format(a/100.)
'1.795200000000 '
>>> "{:<20.16f}".format(a/100.0)
'1.7952000000000001 '
Thats why I said don't worry about it... the value you get will depend on the machine architecture you have as well.