Geometry precision issue when using SDF

[MingHome]

Dear Python and SDF experts,

I encountered an issue when using SDF/SeDF. 

I have two supposedly identical datasets in both local FGDB (featureclass) and AGOL (hosted featurelayer).  To keep both datasets in sync, I am using Pandas and Spatial DataFrame to compare two dataset and to identify any difference in attributes and/or geometry/Shape. If the changes are detected in source dataset, I will modify the target dataset accordingly. It is tested great for all the attributes. However, there is an issue to compare the geometry/SHAPE field between two "supposedly identical" SDF created from gdb and agol.

I found that the precision of coordinators would be altered after a feature is copied/published from FGDB to AGOL, or vice versa. Although the difference is tinny, 9th or 10th digits after decimal, it failed to pass the "identical test" with SDF merge operation. Below is a example:

SHAPE column of Line feature created in FGDB: 

    {'paths': [[[1371391.9402663112, 549074.1051290631], [1371627.089730382, 547898.3574805558], ... ...

Copied to AGOL:

    {'paths': [[[1371391.94026631, 549074.105129063], [1371627.08973038, 547898.357480556], ... ...

Does any one encountered similar issue? and know how to resolve this issue?

I thought of using the Pandas.dataframe.round("SHAPE",8) function to round the decimal of the SHAPE coodinators to 8 or even less. However, it seem that the SHAPE in sdf is not a "number" type and the round would not work if I direct use the round function against "SHAPE". Any thoughts and suggestions?

Thanks,

Ming

[jcarlson]

At a glance, I would assume that the two data sources have different spatial resolutions?

You should be able to load both layers into Pro and check their spatial references there. You can also go to rest-service-url/0?f=json and look for the resolution in the properties listed.

[MingHome]

It seems that the spatial resolution and precision settings are the same, they should be set to double-precision. In fact, I published the featureclass from fgdb to agol.  However, it seems that the coordinates stored in FGDB is 18 bytes (length in 18 digits, including the decimal), butin agol is 16 bytes for polyline (18 bytes for points!). I am not for sure if there is a way to set the coordinates to 16 bytes in fgdb.

Thanks for reply!

Ming

[jcarlson]

I can't see a way around this...

In our process of updating layers back and forth, we just use the "last updated" editor tracking timestamp as a signal to update all attributes and geometry, since there wasn't a reliably consistent way to pick out the true differences in the shape field.

[MingHome]

Thanks for the tip!

The xy resolution of both dataset is exactly the same: 0.000328083333333333 US Survey Feet

Mng

[DanPatterson]

 is the coordinate system of both the same? and if so, what is it?

[MingHome]

Yes, both is 'spatialReference': {'wkid': 102723, 'latestWkid': 3735}

[MingHome]

Found a similar post on stackexchange.com. However, it uses geopandas and shapely which I am not familiar with. In fact, I am fairly new to Python and ArcGIS API too! I guess the solution may be applied to pandas.DataFrame.spatial too. Any suggestions or help is appreciated!

Ming

[DanPatterson]

shapely is used by both (Geopandas is built on top of Pandas)

if arcpy isn't used for geometry shapely is used for the geometry stuff

[MingHome · Accepted solution]

Finally, I found solution to this issue!

In this case, what we care is really NOT the precision of the coordinates, rather, is compare value with limited precision (8 decimals should work). Therefore, I added a new column with the string type of the “SHAPE data” in both sdf. Then, use the “regular expressions” to round all the numbers in string to 8 decimals. Finally use the “new string column” for “merge operation”. It works!

Thanks for Josh Carlson and Dan for you inputs!

Ming