topic Re: Problem with precision and scale in new field in Data Management Questions

Problem with precision and scale in new field

Md_YousufReja — Mon, 18 Jul 2016 19:28:13 GMT

Hello GeoNet community,

I have a question regarding precision and scale of a new field. I am working on making a new field where I need total of 18 digits including 8 digits after decimal. For example the math is

900000000 + 1111*10000 + 1111*1 + 11111111/100000000

I need the result is 911111111.11111111

But, it shows 911111111.11111116

FYI, I have used Double and changed the precision and scale when I add the field. I will be appreciated if anyone has some ideas to explore.

Thanks

Reja

Re: Problem with precision and scale in new field

AbdullahAnter — Mon, 18 Jul 2016 20:21:19 GMT

In attribute table

right click on the top of the field that you want to set specific precision . and click on properties

Inside Field Properties window. click on dots icon ( ... ) it is beside Numeric.

choose from The left Numeric, then choose number of decimal places . and set it to be 8 as you want.

Re: Problem with precision and scale in new field

Md_YousufReja — Mon, 18 Jul 2016 20:27:49 GMT

Thanks Anter! I did this part too. However, the problem is still unsolved. I don't know why this is rounding up.

Re: Problem with precision and scale in new field

AbdullahAnter — Mon, 18 Jul 2016 20:32:42 GMT

you said " I need the result is 911111111.11111111

But, it shows 911111111.11111116"

Both precision is 8

The difference in value not in precision.

Re: Problem with precision and scale in new field

Md_YousufReja — Mon, 18 Jul 2016 20:36:57 GMT

No, actually the scale is 8, precision is 17. Do you have any idea then why the value is changed as I used the scale is 8?

Re: Problem with precision and scale in new field

RichardFairhurst — Mon, 18 Jul 2016 20:38:46 GMT

I believe the maximum number of digits stored internally in a double is 17 digits, so you may have exceeded the limits of what it is capable of displaying reliably. At least this is true in .Net.

System.Double

"Because some numbers cannot be represented exactly as fractional binary values, floating-point numbers can only approximate real numbers.

All floating-point numbers also have a limited number of significant digits, which also determines how accurately a floating-point value approximates a real number. A Double value has up to 15 decimal digits of precision, although a maximum of 17 digits is maintained internally. "

Re: Problem with precision and scale in new field

AbdullahAnter — Mon, 18 Jul 2016 20:46:19 GMT

Yusuf, How did you change the scale?

Re: Problem with precision and scale in new field

AbdullahAnter — Mon, 18 Jul 2016 20:49:34 GMT

I calculate the same Yusuf math in my calculator

the result not exceed 8 decimal digits of precision.

the result is 911111111.11111111

Re: Problem with precision and scale in new field

DanPatterson_Retired — Mon, 18 Jul 2016 20:55:15 GMT

You have reached the limits of floating point precision. The only other build in math module that support higher representation is Python's Decimal class and I think the mpmath module... both of which can't be used in esri products. You have an alternate solution if this is purely for display purposes... that is to concatenate string representations of the integer and decimal portions of the number. Your number is being represented as best it can and won't affect calculations

91111111.1111111

>>> a = 911111111.11111111

>>> a

911111111.1111112

>>> a-a

0.0

Re: Problem with precision and scale in new field

AbdullahAnter — Mon, 18 Jul 2016 21:11:24 GMT

Dan, your viewpoint might to be right, I also thought about that .

But

Md Yousuf Reja, wrote in the end of result number 6 not 2.

mdyousufreja wrote:
Hello GeoNet community,
I have a question regarding precision and scale of a new field. I am working on making a new field where I need total of 18 digits including 8 digits after decimal. For example the math is
900000000 + 1111*10000 + 1111*1 + 11111111/100000000
I need the result is 911111111.11111111
But, it shows 911111111.11111116

Re: Problem with precision and scale in new field

DanPatterson_Retired — Mon, 18 Jul 2016 21:13:03 GMT

6 .... 2 I am using python 3.5 as a demo... maybe mine is more 'incorrectly accurate '

Re: Problem with precision and scale in new field

AbdullahAnter — Mon, 18 Jul 2016 21:23:14 GMT

Dan, 6 is not logical

I think there is a different problem not precision.

or Yusuf maybe wrote a wrong number.

Re: Problem with precision and scale in new field

DanPatterson_Retired — Mon, 18 Jul 2016 21:34:53 GMT

get him to do the test.... subtract the two numbers... in the big picture I am sure it won't matter since if he is using floating point representations in that manner, then the problem should be scaled by moving the data towards the 0,0 origin by subtracting the mean from all values... or work with integers... or one of the other ways of dealing with floating point math.

Re: Problem with precision and scale in new field

Md_YousufReja — Mon, 18 Jul 2016 22:03:47 GMT

Thanks Dana And Anter for going deep into it. I am working on making 2,880,000 unique cell IDs for a panel. When I did join the dbf file of all IDs and math, I found that last single digits or sometimes last two digits of values are changing. When I add the field, I used Double as field type and put 17 for precision and 8 for scale. Here are some examples:

900000000 + 2252* 10000+ 1245*1 + 11111111/100000000 = 922521245.11111116

900000000 + 2252* 10000+ 1245*1 + 11111121/100000000 = 922521245.11111116

900000000 + 2252* 10000+ 1245*1 + 11112111/100000000 = 922521245.11112106

900000000 + 2252* 10000+ 1245*1 + 11112121/100000000 = 922521245.11112118

................................................................................like that

Thanks

Re: Problem with precision and scale in new field

DanPatterson_Retired — Mon, 18 Jul 2016 23:43:14 GMT

don't use a dbase file

unique values should be integer, don't rely on float

or scale your unique values as an order selection from system minimum to system maximum

perhaps if you elaborated why you need 2,880,000 unique cell ids for a panel (of what?) it might help come up with some other solutions

Re: Problem with precision and scale in new field

Md_YousufReja — Tue, 19 Jul 2016 14:57:13 GMT

Thanks Dan! I need to develop 2,880,000 cell IDs for a mapping application model, where each cell has to be unique. Because other raster and data layers will be analyzed by the cells. I know that maximum number of digits stored internally in a double is 17 digits. In this case I'm using the max number, but somehow it rounded up the last one or last two digits. Is there anything that I have to fix in the settings?

Re: Problem with precision and scale in new field

DanPatterson_Retired — Tue, 19 Jul 2016 15:15:12 GMT

64 bit system, integer range

>>> min_int = -2**63

>>> max_int = 2**63 -1

>>> print(" from {} to {}".format(min_int,max_int))

from -9223372036854775808 to 9223372036854775807

32 bit system, integer range

>>> min_int = -2**31

>>> max_int = 2**31 -1

>>> print(" from {} to {}".format(min_int,max_int))

from -2147483648 to 2147483647

I think you have enough unique values by sticking to integer