topic Re: Float field value in scientific notation returns incorrectly in Data Management Questions

Float field value in scientific notation returns incorrectly

JeffReinhart2 — Fri, 28 Dec 2012 16:31:00 GMT

I have a value stored in a field (type "float") that is not returning correctly when checked using Arc software (ArcInfo 10.0, sp5). For instance, the following screen shot shows when statistics are run on the AREA field that contains the errant value. Note that all values shown do not have digits in the tens or hundreds places, yet the Sum for the field comes out with a 4 in the tens place.
[ATTACH=CONFIG]20272[/ATTACH]

The AREA value at issue is for the row "VALUE" = 90. If the value is accessed using a SearchCursor, the mysterious 4 shows up. Here are the results of a SearchCursor on the VALUE and AREA fields:

Running script GetValueFail...
11 348300.0
21 6800400.0
22 1271700.0
23 153000.0
24 55800.0
41 37431000.0
42 3447000.0
43 1559700.0
52 5323500.0
71 686700.0
81 18106200.0
82 4005900.0
90 80477104.0
95 50004900.0
Completed script GetValueFail...
Succeeded at Fri Dec 28 12:12:36 2012 (Elapsed Time: 0.00 seconds)

Note that while the field shows 8.04771E+07 in ArcCatalog (which would be the correct value), the value that is returned is 80477104.0.

When the table is exported to dbase and viewed in Excel, or if the value is copied from the field and pasted to different software, the value is correct. However, when the value is checked within Arc, the incorrect value is returned. I have also tested this in ArcMap using the Select by Attributes tool.

The attached zipped folder contains a geodatabase with the table and a toolbox with the script added, the script, the script message, and the exported dbase table.

Why is this value not being returned correctly by Arc software?

Thanks!

Re: Float field value in scientific notation returns incorrectly

Anonymous User — Fri, 28 Dec 2012 16:56:24 GMT

Original User: vangelo

The answer is accurate, it's just not precise.

4-byte IEEE floating-point values can't store more than seven significant digits --
the rest is rounding error. If you need more significant digits, use the 8-byte
"double" IEEE floating-point datatype.

- V

Re: Float field value in scientific notation returns incorrectly

JeffReinhart2 — Fri, 28 Dec 2012 17:45:29 GMT

That makes sense. To go further on the issue, this table was generated from the Zonal Statistics as Table tool. How can I get that tool to calculate area to a higher precision field type?

Re: Float field value in scientific notation returns incorrectly

Anonymous User — Fri, 28 Dec 2012 17:49:39 GMT

Original User: jeff.m.reinhart

Also, why would it randomly return a 4 and not just a zero?

Re: Float field value in scientific notation returns incorrectly

Anonymous User — Fri, 28 Dec 2012 18:29:48 GMT

Original User: mkennedy

It looks like the tool will only calculate integers or float values, possibly because the input rasters are either integer or float: How zonal statistics works.

Melita

Re: Float field value in scientific notation returns incorrectly

JeffReinhart2 — Fri, 28 Dec 2012 18:47:42 GMT

Seems to me that since the AREA field is derived from the COUNT field, and would normally result in a much larger number (for instance, a 30x30 meter raster would be COUNT * 900), that the AREA field would more appropriate as a type "DOUBLE".

For now, I just worked around it by deleting the AREA field, adding it back as a type "DOUBLE", and calculating it using the appropriate conversion from the COUNT field.

Thanks for the responses!

Re: Float field value in scientific notation returns incorrectly

Anonymous User — Sun, 12 Dec 2021 04:33:12 GMT

Original User: vangelo

The "4" isn't random, it's just a remainder. Wikipedia does a better job of describing IEEE
encoding than I can, but at it's core, the encoding divides the mantissa by 2 repeatedly,
storing the 1 or 0, and carrying the rest. When the manitssa is full, it stops. This has the
effect of truncating the value down, so when it's decoded into a double (which is used for
formatting), the extra bits expand to fill the larger mantissa (resulting in a value slightly
smaller than the original value).

I don't know if you can tell the stats function to use double values in the table, but you
could reduce the magnitude of the values (change units by 1000) or massage the results
to truncate insigificant digits using a base10 logarithm --

lop = log10(value) - 6;
factor = pow( 10, lop);
value = trunc( value / factor ) * factor;

(or something close -- I didn't check the math).

- V

Re: Float field value in scientific notation returns incorrectly

JeffReinhart2 — Fri, 28 Dec 2012 18:58:39 GMT

Very interesting response on where the 4 came from! Thanks!

Here's to another workaround! 🙂