nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Attribute Table with Double Field set to 1 Decimal Place has Inconsistent Rounding

Melissa_B

In a feature class with a double field, some values are not rounding correctly when set to a reduced decimal place using the number format > Rounding > decimal places.

In the attached image, I have the same data in both fields. One is set to the default 6 decimal places. One is set to 1 decimal place. Why is 1.15 shown as 1.1 but 1.55 shown as 1.6? Why is 1.45 shown as 1.4 but 4.45 shown as 4.5?!

I've submitted this to ESRI support since it's happening on multiple computers, multiple ArcGIS Pro versions (3.4.3 and 3.5.2), different projects, and different feature classes, but I'm still over here confused like, has anyone else noticed this?! I came across the issue because a dynamic text in a layout was not rounding some values correctly when set to one decimal place in the formatting. Then I was like "well what happens to an attribute table set to one decimal place?" The answer is the same behavior! Am I crazy? Is there some secret rule to rounding numbers I never learned? Do I have two computers with the same CPU error?

inconsistent rounding.png

Brian_McLeer_0-1741304632006.png

Find more posts tagged with

Accepted answers

NicoleJohnson

I'm not a math person but my spouse is, so I asked him about this. It has to do with floating point number representation.

Try entering 1.15 as a double here and press enter: https://float.exposed/0x3ff2666666666666

You'll see that the result is 1.14999 (it continues, but you get the idea), which rounds down to 1.1 with a precision of 1.

For more info you can see https://en.m.wikipedia.org/wiki/IEEE_754. I can't say I understand it, but it's why you're seeing the results you're seeing.

Edit: here's a much more readable explanation than the Wikipedia article: https://spin.atomicobject.com/floating-point-numbers/

All comments

Melissa_B

I think maybe people tried to report this at some point and it was dismissed:

https://support.esri.com/en-us/bug/numerical-values-are-rounded-incorrectly-when-using-arc-bug-000115244

https://support.esri.com/en-us/bug/an-arcade-label-expression-does-not-round-a-decimal-num-bug-000104469

Because if you have the field set to 1 decimal place in the formatting for the attribute table and you use any other language but Arcade to label, it will match what the attribute table shows. But Arcade will not honor the formatting of the table so the whole value shows. But if you round with the Arcade rounding function, it rounds correctly and DOESN'T MATCH WHAT'S SHOWN IN THE ATTRIBUTE TABLE.

Melissa_B

If someone out there could please test this on their system using a basic feature class with a double field and the following values, I would be most grateful:

1.15

1.25

1.45

1.55

1.65

1.75

2.15

2.65

2.55

Is this a literal glitch in the matrix?

Melissa_B

I thought maybe the rounding operation was following the rule of rounding to the nearest even value instead of just rounding up for all values ending in 5, but then why does 1.45 go to 1.4 but 4.45 go to 4.5?

Is it random rounding? Does anyone know what kind of rounding operation ArcGIS Pro uses? I can't find anything when I try to look through the documentation.

Apparently some programming languages use a different rounding method for decimal values - does anyone know what language ArcGIS Pro is programmed in?

Melissa_B

In case anyone in the distant future finds this post, for the dynamic text at least, I'm using the following Arcade function to make sure it's rounding UP to one decimal place: TEXT(ROUND($feature.field,1),"0.0") I assume all of the decimal place rounding options are random but Arcade's round function is documented as rounding up. I'm not seeing how to change that in the attribute table, but at least layouts will show numbers as they need to be for our use case.

If anyone knows of some setting that can be changed to change that rounding algorithm, it would save me a lot of time adjusting layouts. I can't find documentation of a setting like that when I try to search for it.

Melissa_B

I did a test and got these results:

full value - 1 decimal place - round direction

1.05 - 1.1 - up
1.15 - 1.1 - down
1.25 - 1.3 - up
1.35 - 1.4 - up
1.45 - 1.4 - down
1.55 - 1.6 - up
1.65 - 1.6 - down
1.75 - 1.8 - up
1.85 - 1.9 - up
1.95 - 1.9 - down

So it rounded up 6 times and down 4 times. But that doesn't even resolve rounding bias?

And it's not consistent:

2.05 - 2.0 - down
2.15 - 2.1 - down
2.25 - 2.3 - up
2.35 - 2.4 - up
2.45 - 2.5 - up
2.55 - 2.5 - down
2.65 - 2.6 - down
2.75 - 2.8 - up
2.85 - 2.9 - up
2.95 - 3.0 - up

Here it rounded up 6 times and down 4 times again but it didn't follow the same pattern. Am I missing something obvious here?

JoshuaBixby

What you are seeing is consistent if you understand the issues with floating point math, including rounding. There is a floating point standard (IEEE 754) that is broadly adopted and provides deterministic handling of floating point storage and arithmetic. One of the big challenges with floating points is that many/most base-10 decimal numbers are not stored exactly but as approximations. The more bytes available to store the number, the closer the approximation is to the actual number. There is a lot written about this topic, and what you can find in a search will do better at explaining it than I can here.

In addition to the floating point approximation issue, there is no single way to round. Not all programming languages implement the same default rounding rule, so it is possible the same floating point number can be rounded differently between different languages. The Wikipedia page on rounding does a fair job of summarizing the situation with rounding numbers.

NicoleJohnson

I'm not a math person but my spouse is, so I asked him about this. It has to do with floating point number representation.

Try entering 1.15 as a double here and press enter: https://float.exposed/0x3ff2666666666666

You'll see that the result is 1.14999 (it continues, but you get the idea), which rounds down to 1.1 with a precision of 1.

For more info you can see https://en.m.wikipedia.org/wiki/IEEE_754. I can't say I understand it, but it's why you're seeing the results you're seeing.

Edit: here's a much more readable explanation than the Wikipedia article: https://spin.atomicobject.com/floating-point-numbers/

Melissa_B

So if this is a well known limitation and there are programmatic ways of reducing the rounding error with floating point values, why does ArcGIS Pro not use those methods in the attribute table or in the dynamic text? Is ArcGIS Online using a language or method to address the floating point problem? Because when I test 0.15 in ArcGIS Online using 1 decimal place it gets rounded to 0.2 but ArcGIS Pro shows it as 0.1. So if it's being handled in ArcGIS Online and by labels with Arcade, then why is it not being handled in ArcGIS Pro for the attribute table and dynamic text?

Since there are so many kinds of rounding, why does ESRI's documentation only say "rounding"? Why do they not specify what method they use? When I see "rounding", I assume it means the common round up method.

Why does ESRI have a support article telling users to set the decimal places in the attribute table but then use Arcade's rounding function for labels when Arcade handles the floating point problem for rounding and is documented to use the round up method? This produces a difference in the values shown in the table versus what is being labeled in maps.

https://support.esri.com/en-us/knowledge-base/how-to-specify-decimal-places-in-attribute-tables-and-l-000025170

Melissa_B

Thank you for those resources! It's good to know that it's not a random rounding algorithm that isn't documented that's being used! Of course, now I have to wonder why it's not being addressed in the ArcGIS Pro user interface so that the values being displayed better match the actual stored values.

Melissa_B

ESRI support has created a bug for this issue: BUG-000178046

NicoleJohnson

@Melissa_B, could you post a link to the bug page? I'm not having any luck searching the bug number (because apparently Esri doesn't let you do that...? 🙄).

I'm curious because it just doesn't seem like a bug, it's a fundamental aspect of what a float is and how it can be stored.

JoshuaBixby

The Esri Support website does allow and work on searching bug numbers. In this case, your search is not finding the defect because Esri has not publicly released the defect. Defects logged through Esri Support do not automatically get made public, and some never get made public although most do. I would wait another 2-4 weeks and check again.

NicoleJohnson

Ohh, that makes sense, thank you!

Edit: I just remembered I tried searching a publicly available bug number earlier to see if it worked, and it seems to just go to a blank "search results" page. Have you tried this recently? Maybe the site is malfunctioning?

Melissa_B

Public bugs can be searched by number on https://support.esri.com/en-us/search

Not all bugs are on the public list though; I think they have to be reviewed first, but I am also not super clear on what gets a bug or enhancement moved to the public list.

The attribute table and dynamic text "decimalPlaces" property only changes how the values appear to users - not how they are stored. So if they are only changing what is displayed to users by using a basic round function without accounting for the floating point problem, then users will get unexpected results; like when I had a polygon recorded as 1.15 acres but was showing as 1.1 acres when set to one decimal place in the dynamic text tag for a map. In my use case, I need the "rounded" values to be displaying the results of a typical round up operation and not a literal round on a floating point approximation.

The Arcade round function addresses the floating point problem - that's why there's a mismatch between what gets labelled using Arcade round() and what gets shown in the attribute table with the same number of decimal places. Additionally, whatever language/bit of code is handling the decimalPlaces property in ArcGIS Online is also handling the issue and rounding up as you would expect. The inconsistencies between label and attribute table and the operation of the "decimalPlaces" property between ArcGIS Pro and ArcGIS Online very much seem like a bug to me.

There are several programmatic methods to "round" a floating point value to display to users of a software that better matches the intent behind a stored value. Several methods are outlined here for C++: https://java2blog.com/round-to-2-decimal-places-cpp/

Also, if the attribute table were consistently being true to floating point values, then when I entered 1.15 into a field set for 6 decimal places, it should show me 1.149999. Instead, the attribute table is displaying 1.15 - so there is already some level of operation being performed when it is showing the "full" value in the table to hide the approximation. Why hide the approximation in the table set to more decimal places than intended but not address that same issue when "rounding" to fewer decimal places?

I've learned my lesson though when I had to change 246 dynamic text tags in 82 layouts to use the Arcade round function instead of the decimalPlaces property. (yes, I desperately wish these layouts could be a map series but there's no effective method I've found for placing insets within a map series that doesn't cause some amount of manual work for each map)

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.