topic Re: append to agol feature service not working in ArcGIS Pro Questions

append to agol feature service not working

Anonymous User — Thu, 30 Mar 2017 21:01:24 GMT

Hi all.

For some reason, I can only append one feature at a time into an existing AGOL feature service.

I bring the feature service layer into Pro, plus a local data source (typically a file geodatabase). I then run the Append gp tool, mapping the fields appropriately. It appends the first feature, then fails with the following message:

Start Time: Friday, 31 March 2017 9:41:20 AM
ERROR 999999: Error executing function.
General function failure [Cannot insert duplicate key row in object 'user_2456.BOAT_SURVEY_DEMO_FISHSPECIES' with unique index 'GlobalID_Index'. The duplicate key value is (9bc2a5e1-d2e2-42fd-9802-7caa78b1c739).
The statement has been terminated.]
Failed to execute (Append).

At first, my local data had no global ids, so I tried adding them, but no change, even if I ticked 'preserve global ids' in the append tool (although that failed with a different message).

Has anyone else experienced this, know what is going on, and/or have a fix?

Cheers,

-Paul

Re: append to agol feature service not working

RussellBrennan — Thu, 30 Mar 2017 22:27:22 GMT

Hi Paul,

This is a bug that was introduced at Pro 1.4. We are investigating the cause of it now and plan to have a fix for this in the next release. If our investigations reveal a workaround I will post it here.

- Russell

Re: append to agol feature service not working

Anonymous User — Fri, 31 Mar 2017 00:20:07 GMT

Thanks Russell.

Re: append to agol feature service not working

MatthewAyer — Tue, 11 Apr 2017 11:21:44 GMT

Are there any suggested workarounds to this bug? We would like to add features to a hosted feature service used with collector, and are now faced with this issue as well.

Re: append to agol feature service not working

Anonymous User — Tue, 11 Apr 2017 11:30:21 GMT

The append works if only one item is selected - I had approx 10 records to append so just did them one at a time.

Assuming this is not a feasible option for large datasets, someone suggested that copy and paste might still work. I gather it just requires the source and target datasets to be exactly the same. I haven't tested this though. To do this, I would download the source feature layer from agol as a file geodatabase, truncate it to remove the existing data, append into the local layer, then try a copy paste into the agol online feature layer.

Let us know if you get that working...??

I'm still hoping we get an actual fix asap!

Re: append to agol feature service not working

DamonEisenach — Fri, 26 May 2017 14:32:05 GMT

I have come up with a way to get the append on repeat to run via a model that keeps running until all items are appended one by one. Only problem is it takes about 4 mins per append and there are usually 100+/- a week.

Re: append to agol feature service not working

CollinMcCormick1 — Fri, 08 Dec 2017 17:39:40 GMT

I look forward to seeing this bug fixed. There is a layer I'd like many to be able to update but it's not reasonable for me to expect them to be willing to select the individual records and update one at a time. I may consider the model approach that damoneisenach‌ came up with. Thanks!

Re: append to agol feature service not working

RussellBrennan — Fri, 08 Dec 2017 21:53:21 GMT

Collin,

This issue was resolved with Pro 2.0. There is also a new Append option on the item page in the latest release of ArcGIS Online. This is a very efficient way to load data into your hosted services. I would recommend looking into this if you are using AGOL hosted feature services.

Russell

Re: append to agol feature service not working

Anonymous User — Sat, 09 Dec 2017 00:36:54 GMT

Hi Collin and Russell

Yes I can confirm the issue (at least in our testing) has been resolved in latest Pro.

I note though that appending (or deleting) large datasets via Pro seems excruciatingly slow to the point where I've found it impossible to update datasets with several thousand features. I gave up using Pro for that and instead used the ArcGIS Python API. I put together a script that reads a source spreadsheet in a panda df, converts to correct json format, then uploads in chunks of about 2000 features at a time. As a comparison, easily an hour or two to append via Pro, about 2 min via the script for a spreadsheet of about 20,000 records.

Note that I don't think the new option to load data direct from the AGOL will work as I think that only works on particular layers, e.g. layers published directly from csv or shape files?? Not sure on that, so something I will have to look into.

cheers,

-Paul.

p.s. See my other post which has had no replies as yet.

Why is ArcGIS Pro so slow to delete features or append features into ArcGIS Online?

Re: append to agol feature service not working

DrewFlater — Thu, 21 Dec 2017 07:33:53 GMT

Hi Paul, can you share some code snippets of using the Python API to load data into your feature service?

Re: append to agol feature service not working

Anonymous User — Thu, 21 Dec 2017 21:34:03 GMT

Hi Drew

Below is the basic script I put together. It's pretty straightforward.

Just be warned that, if run as is, it truncates the dataset and deletes everything! Don't get caught out if that's not what you want to do!

You would need to adjust the part that reads the source data and manipulates it into the json object as you need to construct an object with the correct fieldnames etc. I was just updating plain tables, so no geometry involved here but could be worked in I guess.

from arcgis.gis import GIS
from getpass import getpass #to accept passwords in an interactive fashion
import pandas as pd
import time
import datetime

username = input()
password = getpass()
gis = GIS("https://www.arcgis.com", username, password)

item_to_update = gis.content.get('item id from agol')

for index, table in enumerate(item_to_update.tables):
if table.properties.name == 'name of the table':
table_to_update = table

#confirmation we have the correct table...
print(str(table_to_update.properties.name))

# WARNING!! This deletes EVERYTHING in the table!
table_to_update.delete_features(where="objectid > 0")

# A couple of little helper functions:

def getDate(datestring):
if datestring is None:
return None
datestring = str(datestring).strip()
if datestring == 'nan' or datestring == 'NaT':
return None
try:
return datetime.datetime.strptime(datestring, "%Y-%m-%d %H:%M:%S")
except:
pass
return None

def getString(s):
if s is None:
return ''
if isinstance(s, str) is False:
s = str(s).strip()
if s == 'nan':
return ''
return s

sourceDataFrame = pd.read_excel(r'c:/temp/spreadsheet_data.xlsx')
sourceDataFrame.head()

features_to_be_added = []
notifier = 0

#this bit needs to be customised for the data source and data it contains...
for index, row in sourceDataFrame.iterrows():
notifier += 1
if notifier == 1000:
print('Number processed: ' + str(index + 1))
notifier = 0
try:
nextFeature = {"attributes":{}}
nextFeature["attributes"]["field1"] = getString(row['Field1'])
nextFeature["attributes"]["field2"] = getString(row['Field2'])
nextFeature["attributes"]["field3"] = getDate(row['Field3'])
features_to_be_added.append(nextFeature)
except ():
print(">>>>>>>>>>>>>>>>>>Error at index: " + str(index) )

len(features_to_be_added) #confirmation of number of features...

start = time.clock()
print('Number of features to upload: ' + str(len(features_to_be_added)))
chunksize = 2000
chunks = [features_to_be_added for x in range(0, len(features_to_be_added), chunksize)]
for index, chunk in enumerate(chunks):
print('chunk #' + str(index)+ ' with ' + str(len(chunk)) +' features started at {0:.2f} seconds'.format(time.clock() - start))
results = table_to_update.edit_features(adds = chunk)
print('Completed upload of {0} features in {1:.2f} seconds'.format(str(len(features_to_be_added)), time.clock() - start) )

Re: append to agol feature service not working

RussellBrennan — Tue, 09 Jan 2018 19:56:24 GMT

Hi Paul,

This is currently a limit with feature services in Pro. In the current software appending is done one row at a time. We are investigating possible performance improvements for the next few releases such as grouping the edits into fewer calls or use the Append API on the back end.

Re: append to agol feature service not working

Anonymous User — Fri, 12 Jan 2018 22:41:26 GMT

Thanks Russell. Good to know.

cheers,

-Paul