Floating Point Rasters and Grid Values

342
2
09-14-2011 01:19 PM
SAMUELAMOAKO-ATTA
New Contributor
I have a floating point grid/raster and will want to make all values rounded to the nearest tenth.

I have converted to integer (added 0.5 for rounding), but the final floating point output grid is not giving me the exact values I want. For e.g. 4676.0000 should become 467.6, but instead I get 467.60006.

Also 4674.00000 should become 467.4 but instead I get 467.399994.

Any help and suggestions?
Thanks
Sam
0 Kudos
2 Replies
Luke_Pinner
MVP Regular Contributor
Have a read of the floating point tutorial. It's for python, but don't worry about that, it's one of the best explanations of the floating point representation issue I've come across.

Basically, don't worry about exact values, what you're getting is as close as can be represented with as base 2 (binary) fractions. If you really must have control over this, keep your data as integer (multiply by 10^n where n is the number of decimal places you wish to keep).
0 Kudos
SAMUELAMOAKO-ATTA
New Contributor
I'll try that and see. Thanks for the link on floating points.
0 Kudos