Rounding error when using constant raster

1889
3
02-22-2012 07:38 AM
JohnDoe1
New Contributor
I created a constant raster using the toolbox with value 1 (float)

I have the following python code:
a = (Raster('myRaster') * 1.724)

When I print the maximum/minimum of 'a', I get 1.72399997711

I also have a feeling that these rounding errors are causing some other problems in my script.

How can I fix this without manually rounding every operation?
Tags (2)
0 Kudos
3 Replies
DanPatterson_Retired
MVP Emeritus
do a web search on floating point representation .  There is nothing you can do about it
0 Kudos
JohnDoe1
New Contributor
do a web search on floating point representation .  There is nothing you can do about it


I thought it would be  some numerical error, but I'm wondering why
a = 1.724
b = 1.0 * 1.724
print b #=1.724
does work, and my original code doesn't?
0 Kudos
DanPatterson_Retired
MVP Emeritus
because of floating point representation
0 Kudos