I use a script to get data into our ArcGIS online organization, but it seems like the GeoAccessor function messes with the vertices and outputs wrong geometry.
Source Data:
Polygon after adding to ArcGIS online using the script below:
Not sure how visible it is on the screenshots here, but it seems like the 2 inner holes of this big polygon get all mesed up when converting the geodataframe to a sedf.
This is my script:
import geopandas as gpd
from arcgis import GIS
import requests
from arcgis.features import GeoAccessor
gis=GIS("https://arcgis.com",username, password)
base_url = 'https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/DoD_Sites_Boundaries_Public/FeatureServer/0/query'
payload = {
'f': 'geojson',
'where': "SITE_NAME='Eglin Air Force Base'",
'outSR': 4326,
'outFields': '*'
}
response = requests.get(base_url, params=payload)
gdf=gpd.GeoDataFrame.from_features(response.json()['features'], crs="epsg:4326")
sedf=GeoAccessor.from_geodataframe(gdf, column_name="geometry")
lyr = sedf.spatial.to_featurelayer('test_poly')
I have fiddled with this for hours. I tried the .buffer(0) method - didn't work. I tried the make_valid() function from shapely.validation too. Filling holes before converting to a spatially enabled dataframe worked, but it is not an acceptable solution.
Any ideas? I have thought the vertices ordering might be to blame, since ESRI uses different clockwise/counterclockwise direction for exterior/interior rings, but I don't know how I would go about to change those.
Or maybe there is a way to bypass GeoAccessor altogether when trying to publish / append to a feature layer?
Solved! Go to Solution.
While my Issue on the Esri's Github is still "under consideration" (link here), I have found a workaround that does the job for us. Basically I bypass GeoAccessor.from_geodataframe() function altogether and append the data to an existing feature layer using:
Target_Layer = gis.content.get('{}'.format(layerid))
json_gdf=gdf.to_json()
insights_dict=json.loads(json_gdf)
# create FeatureSet from json dict and append data to the target layer
fset=FeatureSet.from_geojson(insights_dict)
for feature in fset:
add_result = Target_Layer.layers[0].edit_features(adds=[feature])
related? spatial.from_geodataframe() mangles geometries · Issue #931 · Esri/arcgis-python-api (github.com)
you might want to start an "Issue" there to narrow down the possibilities.
Looking more at this, I am pretty sure it's a bug.. Thanks for sending that link! I don't even know if I should be glad or cry, these probably take a while to get solved.
Do you know of any other way to publish a geodataframe as a feature layer?
Have you tried repairing the geometry first? Admittedly I mix the API with ArcPy, but to dissolve groups of polygons I had to run arcpy.management.RepairGeometry (used OGC validation) first.
So I had sqlite - gdf - sedf - fc - repair geoms - dissolve. Maybe not 100% related, but sometimes shifting formats/APIs can bypass the problem.
I tried to repair geometries using shapely functions, thing is, by the time I would be able to use arcpy repair - the geometries are already mangled. But yes, that is my thought too.. I need to get there using different formats/APIs, I am trying to add the data to a regular dataframe instead of geodataframe at the moment, but that gives me some other errors.
While my Issue on the Esri's Github is still "under consideration" (link here), I have found a workaround that does the job for us. Basically I bypass GeoAccessor.from_geodataframe() function altogether and append the data to an existing feature layer using:
Target_Layer = gis.content.get('{}'.format(layerid))
json_gdf=gdf.to_json()
insights_dict=json.loads(json_gdf)
# create FeatureSet from json dict and append data to the target layer
fset=FeatureSet.from_geojson(insights_dict)
for feature in fset:
add_result = Target_Layer.layers[0].edit_features(adds=[feature])
There is also a simpler way of implementing this process. You don't need to use neither Geopandas nor GeoAccessor:
from arcgis.gis import GIS
from arcgis.features import FeatureLayer
gis=GIS("https://arcgis.com",username, password)
Target_Layer = gis.content.get('{}'.format(layerid))
flyr = FeatureLayer("https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/DoD_Sites_Boundaries_Public/FeatureServer/0", gis=gis)
features = flyr.query(where="SITE_NAME='Eglin Air Force Base'")
for feature in features:
add_result = Target_Layer.layers[0].edit_features(adds=[feature])
Cheers
Mehdi
Thank you for sharing this. In reality, my code was much more complicated than the snippet I posted here, and I needed to process large amounts of data, not just copy them from one source to another. The query only works for 2k features and I wasn't able to find a way to buffer a FeatureSet, so I had to use a Geodataframe.. but that's already off topic.
In the meantime, I received an email from the developers that the GeoAccessor bug was fixed.