topic Re: Error with decimal places in Data Management Questions

Error with decimal places

DamianMilne — Thu, 01 Mar 2012 00:04:59 GMT

Hi all, I have a table in a geodatabase. Several of the values in one of my numeric fields is 15.099999999999. I'm want to change it to 15.1. I've tried everything but the value refuses to change. I've tried manual editing, rounding with field calculator, changeing between floating and double formats. Nothing works. Is this a glitch in ArcGIS 10?? Please help. Cheers, Damian

Re: Error with decimal places

AlexeyTereshenkov — Thu, 01 Mar 2012 08:02:00 GMT

Hi Damian,

What is the data type of the field? Float? Do you store this table in a file geodatabase?

I'd suggest to try the following. Create a new field with the Double data type and then use Field Calculator to calculate this field with the values of the one you struggle with. This should give you 15.1 written to the Double field.

What regional settings do you have? ArcMap "likes" English regional settings (go to the Region and Language settings of your operating system to see what symbol is used as the Decimal symbol). What is it?

Re: Error with decimal places

DamianMilne — Thu, 01 Mar 2012 22:44:22 GMT

Hi tereshenkov, thank you for getting back to me.
My field datatype is Double. The table is stored in a file geodatabase. I did exactly as you said, created a new field (type Double) and used field caculator to round the field value. Nothing changed. I have attached a print screen so you can see exactly what I did, as well as my field values (original values 'SpeciesID', calculated values 'SpeciesID3). I re-ran the field calculator with another formular to make sure it was working and it does. My Regional settings are English (Australia) and under 'Customize Regional Options' my decimal symbol is "."

Any other thoughts? Cheers, Damian

Re: Error with decimal places

AlexeyTereshenkov — Fri, 02 Mar 2012 06:25:48 GMT

Hey Damian,

I see. Could you please try to do what I've outlined earlier but in the field calculator window, just use the [SpeciesID] field without any additional calculations? If you've created a field with the default settings, then it should round up this for you.

Re: Error with decimal places

FabianBlau — Fri, 02 Mar 2012 08:30:00 GMT

There seems to be no error at all. The problem is the representation of float/double.
Have a closer look to:

http://en.wikipedia.org/wiki/Floating_point

You will learn, that there is not for all numbers a representation. That means the number 753.1 does not exists for your computer.
You cannot avoid this problem.

Re: Error with decimal places

DamianMilne — Sun, 04 Mar 2012 23:24:41 GMT

Hi tereshenkov, i created the new field (double), with the default settings, and did the calculation you suggested. No joy, still the same issue. Fabl seems to have the correct answer at the moment.
Damian

Re: Error with decimal places

TiffanyPuett — Mon, 26 Mar 2012 18:29:48 GMT

What data format are you working with? SHP, GDB, MDB, SDE, ETC? I ran into a similar issue when calculating records stored in SDE. If you are try running the calc in a gdb or mdb featureclass.

Turned out to be a known issue back then; don't know if the latest sp fixes it or not.

Re: Error with decimal places

DamianMilne — Mon, 26 Mar 2012 22:36:40 GMT

I'm using GDB. I'm still waiting for our department to install the latest service pack to see if that fixes the issue or not.

Re: Error with decimal places

KimOllivier — Tue, 27 Mar 2012 06:55:32 GMT

All you have to do is change the display to round to one decimal place. Leave the values as a double type.

You can right-click on the table column to alter the number of decimal places shown, and the values will be nicely rounded.

As others point out floating numbers are always approximately converted from binary to decimal except for some edge cases where they have exact equivalents.

Some databases support a decimal type where the digits are stored as characters and decimal arithmetic is performed. Only needed for traditional money calculations. I think its a bit odd because even those are rounded if you allow division, but I suppose accountants don't do division! They don't even allow negative numbers.

Re: Error with decimal places

DamianMilne — Tue, 27 Mar 2012 22:37:35 GMT

Changing the display doesn't solve my problem. I need to round the values for analysis purposes. Changing the display doesn't change the underlying value when performing analysis (and yes, i've tried.)

Re: Error with decimal places

KimOllivier — Wed, 28 Mar 2012 08:18:42 GMT

OK, well then just use the built-in rounding functions in Python and do the calculation using an UpdateCursor.
If the round values are a result of floating point storage rather than proper variability then you could use integers.

If you want to do real decimal arithmetic there is a Decimal class in the decimal module.

The way to store the data would be to use integers and an implied decimal point. Then you use integer arithmetic or convert the integers to decimal type to do the arithmetic and then convert them back again to store.

Re: Error with decimal places

DamianMilne — Wed, 28 Mar 2012 23:09:26 GMT

Hi Kim, could I trouble you to give me the nuts & bolts of how to use "intergers with an implied decimal point" and "interger arithmetic...to decimal type...and back again to store" please?

Re: Error with decimal places

KimOllivier — Sat, 11 Dec 2021 22:41:28 GMT

There is a lot of help in the Python documentation on the Decimal module. Numbers are stored as strings and converted to a Decimal datatype.
Basically you avoid binary arithmetic by storing all numbers as decimal strings and using the Decimal datatype which has some basic arithmetic operations.

>>> import decimal
>>> a = decimal.Decimal("1234.12")
>>> be = decimal.Decimal("2.0")
>>> a / be
Decimal('617.06')
>>> x = 1234.12
>>> y = 2.0
>>> x/y
617.05999999999995

If you use integers offset and scaled by the number of decimal places you want to keep, then you can do integer arithmetic and round off correctly.
This is how Esri handles floating point coordinates in SDE because databases do not have a true floating point type, or it is too slow.
The disadvantage is that there are limits to the range (extents) if you only use 32 bit integers and there is still some rounding or 'creep' after processing.
With 'high precision' 64 bit it is no longer an issue for users. For more details look up "Choosing the resolution and domain of a low-precision spatial reference" in the help for obsolete 9.1 formats to see how you will need to set up your scale and precision to do your own arithmetic with integers and convert back and forth. The Decimal module looks more suitable.