Elevations off by -0.0000235

Question

This is a weird one. I'm working on utility pipes, getting elevations and structure tops. I have found that when calculating geometry, or sometimes transferring numbers with the field calculator, the elevation number I expect is off by exactly -0.0000235 feet.

I'm using Virginia State Grid North for the XY coordinates, and NAVD_1988 for the z values. What I'm doing doesn't require a precision of more than 2 decimal places. But I still wonder why this comes up.

Is there some mathematical significance to this number? Have I found the question to life, the universe, and everything?

MelitaKennedy · Answer

I wish I'd seen this earlier. Numbers are stored in the units of the coordinate system, so yes, they can be in English units, smoots, yards (several varieties), and a bunch of others.

AdamKuch · Answer

Thank you, that is an excellent answer!

It's true that the distance is much smaller than the tolerance for my data. I just need to display two decimal places of a foot. Even 0.01 feet is a very small distance compared to the scale of a site plan. But now if anyone else in my org finds this number, I can tell them why.

When I stared seeing this number, I got very curious.

JoeBorgione · Answer

My guess is its the way one processor handles a floating point or double precision number.

That said, if my math is correct, the difference you are seeing is somewhere along the lines of 0.000282 inches: a human hair is something like 0.0035" in diameter.  I think you're good to go....