|
POST
|
Thanks Dan! I need to develop 2,880,000 cell IDs for a mapping application model, where each cell has to be unique. Because other raster and data layers will be analyzed by the cells. I know that maximum number of digits stored internally in a double is 17 digits. In this case I'm using the max number, but somehow it rounded up the last one or last two digits. Is there anything that I have to fix in the settings?
... View more
07-19-2016
07:57 AM
|
0
|
1
|
1114
|
|
POST
|
Thanks Dana And Anter for going deep into it. I am working on making 2,880,000 unique cell IDs for a panel. When I did join the dbf file of all IDs and math, I found that last single digits or sometimes last two digits of values are changing. When I add the field, I used Double as field type and put 17 for precision and 8 for scale. Here are some examples: 900000000 + 2252* 10000+ 1245*1 + 11111111/100000000 = 922521245.11111116 900000000 + 2252* 10000+ 1245*1 + 11111121/100000000 = 922521245.11111116 900000000 + 2252* 10000+ 1245*1 + 11112111/100000000 = 922521245.11112106 900000000 + 2252* 10000+ 1245*1 + 11112121/100000000 = 922521245.11112118 ................................................................................like that Thanks
... View more
07-18-2016
03:03 PM
|
0
|
1
|
1114
|
|
POST
|
No, actually the scale is 8, precision is 17. Do you have any idea then why the value is changed as I used the scale is 8?
... View more
07-18-2016
01:36 PM
|
0
|
0
|
3007
|
|
POST
|
Thanks Anter! I did this part too. However, the problem is still unsolved. I don't know why this is rounding up.
... View more
07-18-2016
01:27 PM
|
0
|
0
|
3007
|
|
POST
|
Hello GeoNet community, I have a question regarding precision and scale of a new field. I am working on making a new field where I need total of 18 digits including 8 digits after decimal. For example the math is 900000000 + 1111*10000 + 1111*1 + 11111111/100000000 I need the result is 911111111.11111111 But, it shows 911111111.11111116 FYI, I have used Double and changed the precision and scale when I add the field. I will be appreciated if anyone has some ideas to explore. Thanks Reja
... View more
07-18-2016
12:28 PM
|
0
|
16
|
6263
|
| Online Status |
Offline
|
| Date Last Visited |
11-11-2020
02:24 AM
|