Attribute Table with Double Field set to 1 Decimal Place has Inconsistent Rounding

[Melissa_B]

In a feature class with a double field, some values are not rounding correctly when set to a reduced decimal place using the number format > Rounding > decimal places.

In the attached image, I have the same data in both fields. One is set to the default 6 decimal places. One is set to 1 decimal place.  Why is 1.15 shown as 1.1 but 1.55 shown as 1.6?  Why is 1.45 shown as 1.4 but 4.45 shown as 4.5?!

I've submitted this to ESRI support since it's happening on multiple computers, multiple ArcGIS Pro versions (3.4.3 and 3.5.2), different projects, and different feature classes, but I'm still over here confused like, has anyone else noticed this?!  I came across the issue because a dynamic text in a layout was not rounding some values correctly when set to one decimal place in the formatting. Then I was like "well what happens to an attribute table set to one decimal place?" The answer is the same behavior!  Am I crazy? Is there some secret rule to rounding numbers I never learned?  Do I have two computers with the same CPU error?

[Melissa_B]

I think maybe people tried to report this at some point and it was dismissed:

https://support.esri.com/en-us/bug/numerical-values-are-rounded-incorrectly-when-using-arc-bug-00011...

https://support.esri.com/en-us/bug/an-arcade-label-expression-does-not-round-a-decimal-num-bug-00010...

Because if you have the field set to 1 decimal place in the formatting for the attribute table and you use any other language but Arcade to label, it will match what the attribute table shows. But Arcade will not honor the formatting of the table so the whole value shows. But if you round with the Arcade rounding function, it rounds correctly and DOESN'T MATCH WHAT'S SHOWN IN THE ATTRIBUTE TABLE.

[Melissa_B]

If someone out there could please test this on their system using a basic feature class with a double field and the following values, I would be most grateful:

1.15

1.25

1.45

1.55

1.65

1.75

2.15

2.65

2.55

Is this a literal glitch in the matrix?

[Melissa_B]

I thought maybe the rounding operation was following the rule of rounding to the nearest even value instead of just rounding up for all values ending in 5, but then why does 1.45 go to 1.4 but 4.45 go to 4.5?  

Is it random rounding? Does anyone know what kind of rounding operation ArcGIS Pro uses? I can't find anything when I try to look through the documentation. 

Apparently some programming languages use a different rounding method for decimal values - does anyone know what language ArcGIS Pro is programmed in?

[Melissa_B]

In case anyone in the distant future finds this post, for the dynamic text at least, I'm using the following Arcade function to make sure it's rounding UP to one decimal place: TEXT(ROUND($feature.field,1),"0.0") I assume all of the decimal place rounding options are random but Arcade's round function is documented as rounding up.  I'm not seeing how to change that in the attribute table, but at least layouts will show numbers as they need to be for our use case.

If anyone knows of some setting that can be changed to change that rounding algorithm, it would save me a lot of time adjusting layouts.  I can't find documentation of a setting like that when I try to search for it.

[Melissa_B]

I did a test and got these results:

full value   -  1 decimal place  -  round direction

1.05  -  1.1 -  up
1.15  -  1.1 -  down
1.25  -  1.3 -  up
1.35  -  1.4 -  up
1.45  -  1.4 -  down
1.55  -  1.6 -  up
1.65  -  1.6 -  down
1.75  -  1.8 -  up
1.85  -  1.9 -  up
1.95  -  1.9 -  down

So it rounded up 6 times and down 4 times. But that doesn't even resolve rounding bias?

And it's not consistent:

2.05  -  2.0 -  down
2.15  -  2.1 -  down
2.25  -  2.3 -  up
2.35  -  2.4 -  up
2.45  -  2.5 -  up
2.55  -  2.5 -  down
2.65  -  2.6 -  down
2.75 -  2.8 -  up
2.85 -  2.9 -  up
2.95 -  3.0 -  up

Here it rounded up 6 times and down 4 times again but it didn't follow the same pattern. Am I missing something obvious here?

[JoshuaBixby]

What you are seeing is consistent if you understand the issues with floating point math, including rounding.  There is a floating point standard (IEEE 754) that is broadly adopted and provides deterministic handling of floating point storage and arithmetic.  One of the big challenges with floating points is that many/most base-10 decimal numbers are not stored exactly but as approximations.  The more bytes available to store the number, the closer the approximation is to the actual number.  There is a lot written about this topic, and what you can find in a search will do better at explaining it than I can here.

In addition to the floating point approximation issue, there is no single way to round.  Not all programming languages implement the same default rounding rule, so it is possible the same floating point number can be rounded differently between different languages.  The Wikipedia page on rounding does a fair job of summarizing the situation with rounding numbers.

[Melissa_B]

So if this is a well known limitation and there are programmatic ways of reducing the rounding error with floating point values, why does ArcGIS Pro not use those methods in the attribute table or in the dynamic text?  Is ArcGIS Online using a language or method to address the floating point problem? Because when I test 0.15 in ArcGIS Online using 1 decimal place it gets rounded to 0.2 but ArcGIS Pro shows it as 0.1.  So if it's being handled in ArcGIS Online and by labels with Arcade, then why is it not being handled in ArcGIS Pro for the attribute table and dynamic text?

Since there are so many kinds of rounding, why does ESRI's documentation only say "rounding"? Why do they not specify what method they use? When I see "rounding", I assume it means the common round up method.

Why does ESRI have a support article telling users to set the decimal places in the attribute table but then use Arcade's rounding function for labels when Arcade handles the floating point problem for rounding and is documented to use the round up method? This produces a difference in the values shown in the table versus what is being labeled in maps.

https://support.esri.com/en-us/knowledge-base/how-to-specify-decimal-places-in-attribute-tables-and-...

[NicoleJohnson · Accepted solution]

I'm not a math person but my spouse is, so I asked him about this. It has to do with floating point number representation. 

Try entering 1.15 as a double here and press enter: https://float.exposed/0x3ff2666666666666

You'll see that the result is 1.14999 (it continues, but you get the idea), which rounds down to 1.1 with a precision of 1.

For more info you can see https://en.m.wikipedia.org/wiki/IEEE_754. I can't say I understand it, but it's why you're seeing the results you're seeing.

Edit: here's a much more readable explanation than the Wikipedia article: https://spin.atomicobject.com/floating-point-numbers/

[Melissa_B]

Thank you for those resources! It's good to know that it's not a random rounding algorithm that isn't documented that's being used!  Of course, now I have to wonder why it's not being addressed in the ArcGIS Pro user interface so that the values being displayed better match the actual stored values.