Error with decimal places

Damian

Hi all, I have a table in a geodatabase. Several of the values in one of my numeric fields is 15.099999999999. I'm want to change it to 15.1. I've tried everything but the value refuses to change. I've tried manual editing, rounding with field calculator, changeing between floating and double formats. Nothing works. Is this a glitch in ArcGIS 10?? Please help. Cheers, Damian

KimOllivier

There is a lot of help in the Python documentation on the Decimal module. Numbers are stored as strings and converted to a Decimal datatype.
Basically you avoid binary arithmetic by storing all numbers as decimal strings and using the Decimal datatype which has some basic arithmetic operations.

>>> import decimal
>>> a = decimal.Decimal("1234.12")
>>> be = decimal.Decimal("2.0")
>>> a / be
Decimal('617.06')
>>> x = 1234.12
>>> y = 2.0
>>> x/y
617.05999999999995

If you use integers offset and scaled by the number of decimal places you want to keep, then you can do integer arithmetic and round off correctly.
This is how Esri handles floating point coordinates in SDE because databases do not have a true floating point type, or it is too slow.
The disadvantage is that there are limits to the range (extents) if you only use 32 bit integers and there is still some rounding or 'creep' after processing.
With 'high precision' 64 bit it is no longer an issue for users. For more details look up "Choosing the resolution and domain of a low-precision spatial reference" in the help for obsolete 9.1 formats to see how you will need to set up your scale and precision to do your own arithmetic with integers and convert back and forth. The Decimal module looks more suitable.