Error appending file geodatabase to a survey123 hosted feature service

[Jelle_Stu_PR]

Hi,

Using the ArcGIS Pro 3.1 python library, I am trying to append data to a survey123 hosted feature service in ArcGIS Enterprise 10.9.1, .

The file geodatabase that I am trying to append has the exact same attributes as the hosted feature service. There is no GlobalID field in the file Geodatabase and it is not necessary for any relations afterwards. It is sufficient that the globalID is added once it the data is appended to the survey123 hosted feature service. 

# Import necessary modules
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection, FeatureLayer
import arcpy, sys
# Get input parameters

# Make GIS connection
gis = GIS(<portalURL>, "", "")

# Get survey123 Feature Service
surveyItem = gis.content.get(portalId)
fs = FeatureLayerCollection.fromitem(surveyItem)

# Get layer within survey123 feature service
lyr = FeatureLayer(fs.layers[0])


# get input features
inputFeatures = <input file geodatabase path>

# Update features in the feature layer
result = lyr.append(source_table_name = inputFeatures, upload_format = "filegdb")

# Print the result
arcpy.AddMessage(result)

When running the above script, the hosted feature service can correctly be found. I do get, however, the following error:

raceback (most recent call last):
  File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 16931, in _hydrate
    self._refresh()
  File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 16905, in _refresh
    raise e
  File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 16894, in _refresh
    dictdata = self._con.post(
  File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1372, in post
    if url.find("://") == -1:
AttributeError: 'FeatureLayer' object has no attribute 'find'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 34, in <module>
    result = lyr.append(source_table_name = inputFeatures, upload_format = "filegdb")
  File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\layer.py", line 2785, in append
    if self.properties.supportsAppend == False:
  File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 16954, in _hydrate
    self._refresh()
  File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 16905, in _refresh
    raise e
  File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 16894, in _refresh
    dictdata = self._con.post(
  File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1372, in post
    if url.find("://") == -1:
AttributeError: 'FeatureLayer' object has no attribute 'find'

I don't understand the cause of this error. Can aybody help me or know how I can append a file geodatabase to a survey123 hosted feature service using python?

best,
Jelle Stuurman

[JakeSkinner]

Hi @Jelle_Stu_PR,

The File Geodatabase will need to be added as an item to Enterprise before you can append it.  See the following document for an example.  

An easier approach would be to use ArcGIS Pro's append tool to append the File Geodatabase feature class to the hosted feature service.

[Jelle_Stu_PR]

Hi Jake,

Thank you for your swift reply and the suggestion which you provided.

Originally I used the append tool from ArcGIS Pro. I have made a model builder in which I append data to the survery123 hosted feature service. I added the hosted feature service to the map and then added it from there to my model. Within model builder the append tool works fine.

Once I export the model builder as a python script and then run it through a python command,  I get the following error about the hosted feature service

ERROR 000732: Target Dataset: Dataset feature service\js3Items Notities Toevoegen\survey does not exist or is not supported

When trying to add the feature service itself into the model builder as input, the append tool does not recognise it as a valid input:

I therefore came to the conclusion that using the regular append tool does not work when you want to update a feature service through python.

Therefore I tried the way explained in my original post. I tried this method trying to mimic the way you would update a feature service through the user interface in Portal:

I believed that updating a feature service with an offline file geodatabse was therefore possible. But maybe through Prtal's user interface, another tool is used on the background than the python REST arcgis.feature append tool. The returned error in my original post now makes sense as this tool only works with online data sources.

My question therefore is how I can append data from an offline source to a hosted feature service using python and outside of ArcGIS Pro. Do I simply create access using GIS(), call the hosted feature service and the use ArcGIS Pro's append tool in python?

Hopefully I have clearly explained the workflow which I want to achieve.

Thank you in advance for your help.

[Jelle_Stu_PR]

After some further research,  came across this post,that uses the edit_features function.

I edited the suggested code in the post to my dataset and I replaced the pandas module with reguar oython dctionaries. I also removed the geometries because I don't have that.

# Import necessary modules
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection, FeatureLayer
import arcpy


# Get input parameters
fgdb = "<path _to_fgdb>"

# Make GIS connection
gis = GIS(<portal_url>, "", "")

# Create a list to store features as dictionaries
features_as_dicts = []

# Use SearchCursor to iterate through the features in the feature class
with arcpy.da.SearchCursor(fgdb, '*') as cursor:
    for row in cursor:
        feature_dict = {
            'created_user': row[3],
            'created_date': row[4],
            'last_edited_user': row[5],
            'last_edited_date': row[6],
            'webapp_id': row[7],
            'webapp_titel': row[8],
            'notities': row[9],
            'opvolging_nodig_input': row[10],
            'details_opvolger': row[11],
            'status_uitfasering': row[12],
            'datum_uitfasering': row[13],
            'portal': row[14]
        }
        features_as_dicts.append(feature_dict)

#search for the hosted feature layer/service
featureLayer_item = gis.content.search('type: "Feature Service" AND title:"<title>"')

#access the item's feature layers
feature_layers = featureLayer_item[0].layers

#query all the features
fset = feature_layers[0].query()

features_to_be_added = []
#template_hostedFeature = {"attributes": {"webapp_id": None, "created_user": None, "status_uitfasering": None, "globalid": None, "details_opvolger": None, "last_edited_user": None, "datum_uitfasering": None, "webapp_titel": None, "last_edited_date": None, "opvolging_nodig_input": None, "created_date": None, "notities": None, "portal": None, "objectid": None}}

for row in features_as_dicts:
    new_feature = {"attributes": {"webapp_id": None, "created_user": None, "status_uitfasering": None, "details_opvolger": None, "last_edited_user": None, "datum_uitfasering": None, "webapp_titel": None, "last_edited_date": None, "opvolging_nodig_input": None, "created_date": None, "notities": None, "portal": None, "objectid": None}}
    # assign the updated values
    new_feature["attributes"]['created_user'] = row['created_user']
    new_feature["attributes"]['created_date'] = row['created_date']
    new_feature["attributes"]['last_edited_user'] = row['last_edited_user']
    new_feature["attributes"]['last_edited_date'] = row['last_edited_date']
    new_feature["attributes"]['webapp_id'] = row['webapp_id']
    new_feature["attributes"]['webapp_titel'] = row['webapp_titel']
    new_feature["attributes"]['notities'] = row['notities']
    new_feature["attributes"]['opvolging_nodig_input'] = row['opvolging_nodig_input']
    new_feature["attributes"]['details_opvolger'] = row['details_opvolger']
    new_feature["attributes"]['status_uitfasering'] = row['status_uitfasering']
    new_feature["attributes"]['datum_uitfasering'] = row['datum_uitfasering']
    new_feature["attributes"]['portal'] = row['portal']  
    
    features_to_be_added.append(new_feature)

feature_layers[0].edit_features(adds = features_to_be_added) 

Script works fine, uti it tries the editfeatures function. None connections seems to be lost after 5 minutes or so ( I have removed the references to the portal url):

Traceback (most recent call last):
File "test.py", line 87, in <module>
feature_layers[0].edit_features(adds = features_to_be_added)
File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\layer.py", line 3408, in edit_features
return self._con.post_multipart(path=edit_url, postdata=params)
File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1244, in post_multipart
raise requests.exceptions.ConnectionError(
requests.exceptions.ConnectionError: A connection error has occurred: HTTPSConnectionPool(host='<portal_url>, port=443): Max retries exceeded with url: /arcgis/rest/services/Hosted/survey123_.../FeatureServer/0/applyEdits (Caused by ProtocolError('Connection aborted.', ConnectionResetError(10054, 'De externe host heeft een verbinding verbroken', None, 10054, None)))

It looks like it is not able to apply edits. I am a portal admin so I should have the correct privilidges. 

[Kepa]

 Hi everybody,

I came accross with this post while trying to figure out something alike with one of our customers. On an Enterprise environment it's not possible to update a hosted layer using a FileGDB (only on AGOL). I attach a screenshot with the append REST endpoint to show it: 

I think that documentation should reflect this clearer between ArcGIS Online and ArcGIS Enterprise.

Regards,

[Jelle_Stu_PR]

thank you for your note Kepa.

is there another format with which we can append in AGE? 

I checked the equivalent rest endpoint in environment and it looks less eleborate.

Maybe AGE 10.9.1 has some limitations in how to append through python? 

[Kepa]

Hi @Jelle_Stu_PR,

Be careful, you are attaching a screenshot showing another REST operation (add features). Append operation is available adding /append to the desired service's layer/table url:

For AE 10.9.1 you can use: CSV, FeautureCollection, GeoJSON (up to 100MB) and zipped Shapefile (apparently with no size limitation):

EDIT: for updates below 250 features you can use edit_features method from Python API

[Jelle_Stu_PR]

Thank you for making me aware that the append tool uses another URL. The append tool does not seem to work even in the portal UI. It looks like some underlying authentication issues. I will continue this issue ESRI support. Should a solution be found, I will post it back here.