Error with decimal places

[DamianMilne]

Hi all, I have a table in a geodatabase. Several of the values in one of my numeric fields is 15.099999999999. I'm want to change it to 15.1. I've tried everything but the value refuses to change. I've tried manual editing, rounding with field calculator, changeing between floating and double formats. Nothing works. Is this a glitch in ArcGIS 10?? Please help. Cheers, Damian

[KimOllivier]

OK, well then just use the built-in rounding functions in Python and do the calculation using an UpdateCursor.
If the round values are a result of floating point storage rather than proper variability then you could use integers.

If you want to do real decimal arithmetic there is a Decimal class in the decimal module.

The way to store the data would be to use integers and an implied decimal point. Then you use integer arithmetic or convert the integers to decimal type to do the arithmetic and then convert them back again to store.

[DamianMilne]

Hi Kim, could I trouble you to give me the nuts & bolts of how to use "intergers with an implied decimal point" and "interger arithmetic...to decimal type...and back again to store" please?

[KimOllivier]

There is a lot of help in the Python documentation on the Decimal module. Numbers are stored as strings and converted to a Decimal datatype.
Basically you avoid binary arithmetic by storing all numbers as decimal strings and using the Decimal datatype which has some basic arithmetic operations.

>>> import decimal
>>> a = decimal.Decimal("1234.12")
>>> be = decimal.Decimal("2.0")
>>> a / be
Decimal('617.06')
>>> x = 1234.12
>>> y = 2.0
>>> x/y
617.05999999999995

If you use integers offset and scaled by the number of decimal places you want to keep, then you can do integer arithmetic and round off correctly.
This is how Esri handles floating point coordinates in SDE because databases do not have a true floating point type, or it is too slow.
The disadvantage is that there are limits to the range (extents) if you only use 32 bit integers and there is still some rounding or 'creep' after processing.
With 'high precision' 64 bit it is no longer an issue for users. For more details look up "Choosing the resolution and domain of a low-precision spatial reference" in the help for obsolete 9.1 formats to see how you will need to set up your scale and precision to do your own arithmetic with integers and convert back and forth. The Decimal module looks more suitable.