topic Re: Elevations off by -0.0000235 in Data Management Questions

Elevations off by -0.0000235

AdamKuch — Wed, 20 Apr 2016 20:08:49 GMT

This is a weird one. I'm working on utility pipes, getting elevations and structure tops. I have found that when calculating geometry, or sometimes transferring numbers with the field calculator, the elevation number I expect is off by exactly -0.0000235 feet.

I'm using Virginia State Grid North for the XY coordinates, and NAVD_1988 for the z values. What I'm doing doesn't require a precision of more than 2 decimal places. But I still wonder why this comes up.

Is there some mathematical significance to this number? Have I found the question to life, the universe, and everything?

Re: Elevations off by -0.0000235

RichardFairhurst — Wed, 20 Apr 2016 20:43:28 GMT

1. Coordinates never get stored in English units, only in meters, so conversions happen.

2. The Z resolution you have chosen allows this level of movement to be considered the same as an exact value.

3. Z Resolution establishes a Z grid and the Z extent sets an upper and lower bound of possible coordinates, and sometimes this makes it impossible to make whole number coordinates. Check and report these settings.

4. The projection chosen for Z may apply some adjustments when combined with the chosen XY projection (although that is a black box to me so I am not sure how they may interact).

5. Base 10 to base 2 conversions.

6. That distance is less than 1/10th the width of an average human hair and is such a small amount of measure variance from your reported benchmarks that it clearly falls within the tolerances of standard Survey procedures.

Re: Elevations off by -0.0000235

DanPatterson_Retired — Wed, 20 Apr 2016 20:44:32 GMT

floating point representation

Using Dr Google or other search engine... several days worth of reading, should you want to go there

Re: Elevations off by -0.0000235

JoeBorgione — Wed, 20 Apr 2016 20:47:00 GMT

My guess is its the way one processor handles a floating point or double precision number.

That said, if my math is correct, the difference you are seeing is somewhere along the lines of 0.000282 inches: a human hair is something like 0.0035" in diameter. I think you're good to go....

Re: Elevations off by -0.0000235

AdamKuch — Wed, 20 Apr 2016 20:56:16 GMT

Thank you, that is an excellent answer!

It's true that the distance is much smaller than the tolerance for my data. I just need to display two decimal places of a foot. Even 0.01 feet is a very small distance compared to the scale of a site plan. But now if anyone else in my org finds this number, I can tell them why.

When I stared seeing this number, I got very curious.

Re: Elevations off by -0.0000235

MelitaKennedy — Wed, 13 Jul 2016 00:02:17 GMT

I wish I'd seen this earlier. Numbers are stored in the units of the coordinate system, so yes, they can be in English units, smoots, yards (several varieties), and a bunch of others.