topic Re: Precision vs. XY Resolution in Geodatabase Questions

Precision vs. XY Resolution

Bud — Fri, 03 Jun 2022 22:19:18 GMT

ArcMap 10.7.1.:

I've created a new point FC from scratch using Catalog (using all the default settings).

The General tab mentions "Storage: High Precision":

Whereas the Domain, Resolution and Tolerance tab mentions "XY Resolution":

What's the difference between "Precision" and "XY Resolution".

Thanks.

Re: Precision vs. XY Resolution

Anonymous User — Fri, 03 Jun 2022 22:37:07 GMT

This article has helped me several times to understand this.

High precision allows you to store coordinates to incredible detail, and as far as I know is the default. In the long ago past, high precision was not available and old datasets had to be migrated to to high precision storage. These days it is just the normal state of affairs - no changes needed.

Re: Precision vs. XY Resolution

DerekLaw — Sat, 04 Jun 2022 02:41:25 GMT

Hi @Bud,

> What's the difference between "Precision" and "XY Resolution"

Back in 2007, I wrote a detailed Esri technical paper that discussed in detail how spatial domains were stored in feature classes in a geodatabase. Please take a look,

Understanding Coordinate Management in the Geodatabase

It might be a little deeper than you wanted to know, but in case you were interested. 🙂

Hope this helps,

Re: Precision vs. XY Resolution

Anonymous User — Mon, 06 Jun 2022 19:45:42 GMT

@DerekLawthank you for this post. The information in your paper is extremely helpful - this is the kind of post that makes the community so valuable.

I see that the paper is dated 2007. Could you please comment what, if any, changes have occurred since then?

Re: Precision vs. XY Resolution

VinceAngelo — Tue, 07 Jun 2022 16:18:25 GMT

No real changes to the geodatabase model infrastructure since then.

Note that "precision" refers to both the vertex equivalence comparison tolerance AND the spatial reference coordinate storage depth (though at this point, *all* spatial references are HIGH precision).

- V

Re: Precision vs. XY Resolution

DerekLaw — Tue, 07 Jun 2022 16:41:29 GMT

Hi @Anonymous User,

> Could you please comment what, if any, changes have occurred since then?

As @VinceAngelo stated, there hasn't been any major changes to the geodatabase data model since the technical paper was published (otherwise, we would have updated it).

Hope this helps,

Re: Precision vs. XY Resolution

JTLancer — Thu, 21 May 2026 22:39:48 GMT

I'm curious about defaults for State Plane cooordinate system. We are standing up a brand new Enterprise GDB and paying close attention to spatial references as our old database from which we are migrating data through various practices over the years have may inconsistencies.

The screenshot below shows the different srid values by using the Create Feature Dataset tool and different results by by using different settings. For xycluster_tol we get differnet values when accepting defualts with no values specified in environments vs providing these values.

My main question is ESRI articles state 0.003281 would be the correct default value to use. If this rounded number is not pvided the default value ends up being 0.00328083333333333. Does this have an effect if we keep the repeating decimals vs rounding to 6 decimal places. I'm leaning toward the rounded numbers as a "master" dataset. The numbers input in the environments affect other things like the xyunits calc.

Also if we leave m and z cluster_tol blank it defaults to .001 in meters rather than feet. In this case we won't be using m and z, but I imagine as a best practice or potential future use of m or z, that we would want to keep the inital spatial reference properties the same. Any thoughts or input would be appreciated.

@DerekLaw @VinceAngelo

Understanding Coordinate Management in the Geodatabase link does not seem to be active any more

@DerekLaw

This article has been a good resource for me as well.

Re: Precision vs. XY Resolution

DerekLaw — Fri, 22 May 2026 00:22:35 GMT

Hi @JTLancer,

> Understanding Coordinate Management in the Geodatabase link does not seem to be active any more

Thanks for bringing the broken link to my attention. I'll try and get it fixed.

Meanwhile, I've attached a copy of the technical paper to this post. I wrote it back in 2007. 😉

Hope this helps,

Re: Precision vs. XY Resolution

VinceAngelo — Fri, 22 May 2026 23:57:04 GMT

I'm not a big fan of coordsys offsets with large numbers of significant digits. 0.003281 would be an *awful* precision to use. I would choose 0.001 or 0.002, since the conversion to integer is "times 1000" or "times 500" (and not "times 304.78512648582749161840902163974". Precision to a thousandth of a foot is gratuitous overkill for all but precision amoeba farming, but it's a nice round number. I try to avoid the default values, because they were chosen to maximize precision, at the cost of compressibility, and I'd rather transfer a BLOB with 278 bytes than one with 40K (your mileage may vary, but you should take your most complex geometries and try them at different scalefactors, to determine what the exact compression loss might be).

- V

Re: Precision vs. XY Resolution

JTLancer — Fri, 05 Jun 2026 22:09:51 GMT

Thank you for the response @VinceAngelo . If I am understanding you right you are recomending using .001 ft would be a good recommended option. I agree that even this seems a bit overkill as our best data are going to be 1in data under the best of conditions for GPS survey (we are not tracking amoebas). Are there other reasons or considerations on why we would want to consider defining resolutions at a finer level than what could be feasibly captured in the real world? It does seem that moving toward a coarser resolution, would in theory reduce resource usage, storage, and data processing times.

My only issue is this is contradictory to what is stated in ESRI's help documention and articles such as the one referenced in this discussion. This has been the basis for the my original comment.

https://www.esri.com/arcgis-blog/products/arcgis-pro/analytics/geoprocessing-resolution-tolerance-and-hair

Re: Precision vs. XY Resolution

VinceAngelo — Tue, 09 Jun 2026 04:13:34 GMT

Ah,... yes. Well, this is more a religious issue than a technical one. I am not among those who value precision because it is possible, rather than needed. I do see significant performance benefits from my somewhat heretical techniques, though not as much in native RDBMS geometry storage, because non-Esri geometry' doesn't see the same storage reduction found in file geodatabase or ST_GEOMETRY. If your data is only accurate to 1-inch, then 1/0.01' or 1/0.005' could be reasonable XY scalefactors,

The general tools don't make it easy to be consistent in non-default coordinate references, but I've found that using a "Transfer" FeatureDataset with the desired coordref constants as an intermediary for creation, then immediately dragging the newly created FCs out of that FDS to be effective. I usually script my creation utilities, which allows for more flexibility in SpatialReference definition and application, but the FDS technique works quite well for ad-hoc geoprocessing outputs.

- V