topic Re: Error code 500 on appending data with REST API to ArcGIS Online table in ArcGIS REST APIs and Services Questions

Error code 500 on appending data with REST API to ArcGIS Online table

mikaël — Mon, 11 Mar 2024 13:06:54 GMT

I have this process that runs daily, where I get some values from an external source, I transform & store them in a local DB. I then match existing values on AGOL to this local table to make sure the objectids are the same (just to make sure everything is in sync) and add new objectids sequentially locally from the last existing one on AGOL. Then I append (with python) this data to my AGOL table (upload_format = 'geojson', upsert = True, edits = the results from a query on my local DB).

This was running fine, every day, until last week, when I started getting an Exception: Unknown Error (Error Code: 500). The sync and changeTracking parameters are deactivated and I've tried deleting and recreating the table (multiple times)...

What to do?

Please help!

EDIT march 11th

Working with ESRI staff on this, will update with developments. Also noticed that I only have this problem with tables, appending to layers is working correctly.

Re: Error code 500 on appending data with REST API to ArcGIS Online table

JeffSilberberg — Tue, 05 Mar 2024 18:32:49 GMT

If it was running fine until last week, I would open a support ticket since AGOL was updated last week and this may be a behavior change or a new bug.

Re: Error code 500 on appending data with REST API to ArcGIS Online table

mikaël — Tue, 05 Mar 2024 18:57:27 GMT

Well that was exactly what I was thinking, since the problems started occuring on the 26th and the "what's new" blog is dated February 28th, but I did not see anything recent for the REST or Python API.

But yeah good idea, I'll send them an email.

Thanks!

Re: Error code 500 on appending data with REST API to ArcGIS Online table

twhammond — Wed, 17 Apr 2024 14:43:41 GMT

Any word on this one yet? I have a similar situation, except I am appending from an existing gdb.

fLyr.append(item_id=fgd_item.id, upload_format="filegdb", upsert=False, field_mappings=[])

I set up the process to run in task scheduler every few minutes for testing purposes to see what kind of errors I would get. The script worked for weeks beforehand, and after setting it up in task scheduler, it worked for a couple days. Now when I run it, I get the same result as you:

Exception: Unknown Error

(Error Code: 500)

Re: Error code 500 on appending data with REST API to ArcGIS Online table

mikaël — Wed, 17 Apr 2024 14:50:30 GMT

Last update was yesterday, an ESRI analyst is still looking into this issue.

I changed my workflow to use edit_features() which is more complicated because I have to check beforehand which features I have to update and which I have to add, which I did not have to do with append(). But at least it's working now.

Re: Error code 500 on appending data with REST API to ArcGIS Online table

twhammond — Wed, 17 Apr 2024 15:09:02 GMT

Thanks, I'll start looking at moving away from append() in the meantime. Hopefully they get to the bottom of it soon.

Re: Error code 500 on appending data with REST API to ArcGIS Online table

mikaël — Wed, 24 Apr 2024 13:05:19 GMT

Here is the latest update:

The outcome of the escalation was Esri Inc. logging the following defect:
BUG-000166918: Error 500 "Unknown error" encountered when overwriting hosted tables with any file type data using ArcGIS API for Python.
A workaround for this issue is to use a File Geodatabase to append data to the table instead of GeoJSON (or CSV).

Re: Error code 500 on appending data with REST API to ArcGIS Online table

twhammond — Wed, 24 Apr 2024 14:23:37 GMT

That's interesting. I got the issue while appending data using a file geodatabase. However, the bug mysteriously disappeared after a couple days and has been working normally ever since. I honestly have no idea what caused it to break, or why I stopped encountering the error.

So I can't say the workaround doesn't work, but I can say as of last Wednesday I was encountering the same error when appending to table from a geodatabase.

Re: Error code 500 on appending data with REST API to ArcGIS Online table

gmaz — Tue, 04 Jun 2024 22:07:29 GMT

Glad ESRI could help a bit. I'm also appending a file gdb into a truncated dataset and getting the Error Code: 500. It doesn't seem to be a work around that actually works around. I've looked into edit_features(), but given the size of this dataset getting overwritten daily (~15,000 records) it seems like append() would/should be the tool for the job. Curious if you've heard anything else from ESRI.

Re: Error code 500 on appending data with REST API to ArcGIS Online table

mikaël — Wed, 05 Jun 2024 12:41:08 GMT

Nope nothing yet. I've started using AGOL regularly 2 years ago and ESRI opened a couple of bugs from my emails.

No changes yet, so you have to be patient or try to workaround a problem or use something else completely 🙂

Re: Error code 500 on appending data with REST API to ArcGIS Online table

gmaz — Wed, 05 Jun 2024 14:00:34 GMT

Ahh...thank you. I also opened a case with ESRI about this. I'm looking into some workarounds and will let you know if anything comes of the case.

Re: Error code 500 on appending data with REST API to ArcGIS Online table

AndyHess — Thu, 12 Sep 2024 15:47:27 GMT

Has there been an update to this? I am trying to truncate and append using a modified version of the script here: https://community.esri.com/t5/arcgis-online-documents/overwrite-arcgis-online-feature-service-using/ta-p/904457 and I am getting the "Error appending features: Error reading features.
(Error Code: 500)" issue. I changed the script to use a shapefile because apparently using a file geodatabase also doesn't work with the arcgis for python api for some unknown reason. (https://community.esri.com/t5/arcgis-api-for-python-questions/problem-update-hosted-feature-in-portal/td-p/1258473). Thanks for any updates.

Re: Error code 500 on appending data with REST API to ArcGIS Online table

mikaël — Thu, 12 Sep 2024 17:36:17 GMT

No update, although you're not getting the same error I was ("Error reading features"). Maybe in your case the problem is not append() ?

Re: Error code 500 on appending data with REST API to ArcGIS Online table

Justin_Greco — Fri, 11 Oct 2024 19:40:22 GMT

I just started noticing this issue with the 500 error code with one of my scripts that uses append(). But after downloading the ZIP file of the FGDB and using the Update Features on the item page, it worked with no issues. So I tried again with the developer tools open and noticed it was sending the append_fields and field_mappings, which I was omitting in Python since they are optional. The append in Python was successful after setting the append_fields and field_mappings. Though it was the append_fields that fixed it, since I tried without field_mappings.

Re: Error code 500 on appending data with REST API to ArcGIS Online table

mikaël — Tue, 29 Oct 2024 13:25:27 GMT

wow thanks, will try it out!

edit: @Justin_Greco finally managed to test your solution. Unfortunately it doesn't work for me.

I created a new empty feature service, added a couple of records manually on AGOL then tried to append features with a python dict (to FeatureSet to FeactureCollection), either to add or update existing ones, attributes and/or geometry. Tried adding/removing/changing parameters field_mappings and append_fields and also played with upsert and update_geometry. Still getting error 500.

Plus, weirder thing is that when I had troubleshooted this problem in march, I was only getting the problem when appending to tables, not layers. But now I'm getting it with this item which is a layer.

Oh well, at least edit_features() works for me and I changed my current flow to use that function instead so it's not too much of a bother anymore, it's just weird (and I would prefer to use append() which is much simpler).

Leaving the question opened in case there's another answer in a not too distant future 🙂

Thanks!

Re: Error code 500 on appending data with REST API to ArcGIS Online table

mikaël — Tue, 19 Nov 2024 21:35:18 GMT

Well, I think I got it.

My usual workflow was getting data from a local (PostgreSQL/PostGIS) DB and transforming that into a geojson and appending to a layer/table with the upload_format='geojson' parameter. For some reason, it stopped working and I created this post.

When I tried again last time (my last post), I was creating a manual dictionary, transforming that into a featureCollection and using the upload_format='featureCollection' and edits=fc parameters, where fc was my feature collection (from_featureset(from_dict(data))).

Upon further testing today for another project, I was looking yet again at the reference, and noticed that under the edits parameters description, it's written: "Only feature collection json is supported". But I was not sending a feature collection json, I was sending the feature collection object. So I modified the parameter to edits=fc.properties and it was successful!!

The edits parameters description also states: "Append supports all format through the upload_id or item_id". But when I was sending my geojson, I was never using those parameters. So I don't know if it changed at some point. So anyway, I'll try again with a geojson with the upload_id or item_id, and if it still fails, I'll just change my DB query to return a feature collection json instead.

There you go! 🎉