ArcPro: calculate geometry delivers inconsistent decimal places

988
4
08-09-2022 07:23 PM
Gabi
by
New Contributor II

**Using ArcGIS Pro 2.9.0:**
I am trying to calculate UTM coordinates in a shapefile to the 6th decimal place. When I use the calculate geomtery tool as follows:
point x-coordinate (or y-coord, both calculations are having the same issue)>Coordinate Format=Same as input>Coordniate System=UTM 14 N (same as data frame)

Some points correctly show up to the 6th decimal place, and others only to the 2nd while some are just whole numbers.

Gabi_0-1660098072362.jpeg

 

My field type is double with Precision and Scale at the default "0". I've tried changing this number and it doesn't seem to fix the issue. I've updated my "Numeric" setting to "Custom" and entered "0.000000" so now the numbers being rounded just show up with 6 zeroes instead of the proper value. I'm thinking Arc is still doing some rounding on the backend.

Gabi_1-1660098097940.jpeg

 

Has anyone encountered a similar problem/see something I'm doing wrong?

Thanks

0 Kudos
4 Replies
DanPatterson
MVP Esteemed Contributor

What are the values supposed to be?

Changing the table display property doesn't affect how geometry is calculated, just how it is displayed


... sort of retired...
0 Kudos
Gabi
by
New Contributor II

Thanks for asking. 

I am not sure the exact value the coordinates should be, but I know the multiple values coming up as whole numbers should have a series of integers in the decimals place. No way my survey points landed perfectly on these whole lat/long values multiple times. It's also suspicious that the points lacking the 6 decimal places are listed as whole numbers in both the lat and long fields- which were calculated separately (northing/easting in my attribute table).

0 Kudos
BarryNorthey
Occasional Contributor III

Try adding new fields as Double --> Precision 18 --> Scale 6 --> calculate and see if that helps.

0 Kudos
Gabi
by
New Contributor II

Thanks for the suggestion.

Unfortunately I am still getting the same results (same points coming up rounded to a whole number) with the new field set to Precision 18, Scale 6.

0 Kudos