I have a value stored in a field (type "float") that is not returning correctly when checked using Arc software (ArcInfo 10.0, sp5). For instance, the following screen shot shows when statistics are run on the AREA field that contains the errant value. Note that all values shown do not have digits in the tens or hundreds places, yet the Sum for the field comes out with a 4 in the tens place. [ATTACH=CONFIG]20272[/ATTACH]
The AREA value at issue is for the row "VALUE" = 90. If the value is accessed using a SearchCursor, the mysterious 4 shows up. Here are the results of a SearchCursor on the VALUE and AREA fields:
Note that while the field shows 8.04771E+07 in ArcCatalog (which would be the correct value), the value that is returned is 80477104.0.
When the table is exported to dbase and viewed in Excel, or if the value is copied from the field and pasted to different software, the value is correct. However, when the value is checked within Arc, the incorrect value is returned. I have also tested this in ArcMap using the Select by Attributes tool.
The attached zipped folder contains a geodatabase with the table and a toolbox with the script added, the script, the script message, and the exported dbase table.
Why is this value not being returned correctly by Arc software?
4-byte IEEE floating-point values can't store more than seven significant digits -- the rest is rounding error. If you need more significant digits, use the 8-byte "double" IEEE floating-point datatype.
Seems to me that since the AREA field is derived from the COUNT field, and would normally result in a much larger number (for instance, a 30x30 meter raster would be COUNT * 900), that the AREA field would more appropriate as a type "DOUBLE".
For now, I just worked around it by deleting the AREA field, adding it back as a type "DOUBLE", and calculating it using the appropriate conversion from the COUNT field.
The "4" isn't random, it's just a remainder. Wikipedia does a better job of describing IEEE encoding than I can, but at it's core, the encoding divides the mantissa by 2 repeatedly, storing the 1 or 0, and carrying the rest. When the manitssa is full, it stops. This has the effect of truncating the value down, so when it's decoded into a double (which is used for formatting), the extra bits expand to fill the larger mantissa (resulting in a value slightly smaller than the original value).
I don't know if you can tell the stats function to use double values in the table, but you could reduce the magnitude of the values (change units by 1000) or massage the results to truncate insigificant digits using a base10 logarithm --
lop = log10(value) - 6; factor = pow( 10, lop); value = trunc( value / factor ) * factor;