AnsweredAssumed Answered

Dataset not rounding correctly

Question asked by MonoSim on Feb 25, 2011
Latest reply on Mar 11, 2011 by curtvprice
Have an interesting case here where I created a float (prec. 3, scale 2) field (shapefile) then calculated a pct between two fields.

When I attempt to select records with less than 0.35, ArcMap returns records with values 0.35 and less (or < .36).

Exported the table to TXT, when I opened the txt in ArcMap I see that both the values greater than 0.3450 rounded up & values less than 0.3549 values rounded down to 0.349999. Besides the fact those values between 0.35 and 0.3549 should be rounded to 0.35, I was surprised to see the extra decimal places.

I've repeated these steps with other datasets and did not come across this problem. Has anyone else experienced this?

Attached are the .dbr and .txt for comparison.

Outcomes