Did the November 2024 AGOL Update Introduce an Overwrite Bug?

[JACooper]

I am having perplexing issues overwriting a feature layer that was, as recently as last week, routinely overwritten with new data without issue as a part of a large body of python code using the arcgis package. The first failure I noticed was Monday 18 November 2024. The last previous success was Tuesday 12 November, the same day of the November 2024 update to ArcGIS Online. I don't know if my overwrite occurred before or after the application of the AGOL update. 

I've debugged a good deal since this popped up, and it is not exclusively a python API issue. I cannot overwrite the feature layer through the GUI either, even with the same dataset I used to successfully overwrite on 12 November or any prior. The inconsistency plus the documentation noting overwrite was included in the update and citations of other bugs with the update makes me wonder if it is something on AGOL's end. I fully acknowledge that the issue could lie with me, but my debugging hasn't lead me to this yet.

The puzzling thing is that this is only affecting one dataset; other data sources, even ones with similar schemas, are overwriting using the same code just fine. I've had no other GUI or python API overwrite problems except for this one feature layer so far. 

I would try more risky delete-then-reload methods, but the hosted feature layer I am trying to overwrite has view layers and downstream layers scaffolded on top of them used in a production dashboard that I would rather not risk destroying. I have also seen a lot of suggestions (see here, here, here) suggesting replacing the overwrite process with a truncate-append method. This is all fine and well, but A) I have complex data pipelines that use Overwrite in a (usually) very stable way that I would like to keep for now, and B) if this is a better way to update schema-consistent datasets, ESRI could make that implementation on their end. If they offer an overwrite option, it seems reasonable to opt to use it as is.

I have attached the GUI screenshot of the error message as well as the text of the python API error messages. I cannot share the data or feature layer itself due to data privacy constraints, but the layer in question is feeding the layers with the child care provider points on this dashboard. I can add the columns and their types too if that is useful. Any help would be welcome; many thanks.

# arcgis==1.8.5.post3, try 1
...
    flcollection.manager.overwrite(csvowriteversion_path)
  File "~\venv\lib\site-packages\arcgis\features\managers.py", line 1785, in overwrite
    published_item = related_data_item.publish(publish_parameters, overwrite=True)
  File "~\venv\lib\site-packages\arcgis\gis\__init__.py", line 10387, in publish
    serviceitem_id = self._check_publish_status(ret, folder)
  File "~\venv\lib\site-packages\arcgis\gis\__init__.py", line 10614, in _check_publish_status
    raise Exception("Job failed.")
Exception: Job failed.
----------------
# arcgis==1.8.5.post3, try 2
Traceback (most recent call last):
  File "~\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3553, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-15-e4b5a6775c63>", line 1, in <module>
    xstcsv.publish(overwrite=True)
  File "~\venv\lib\site-packages\arcgis\gis\__init__.py", line 10387, in publish
    serviceitem_id = self._check_publish_status(ret, folder)
  File "~\venv\lib\site-packages\arcgis\gis\__init__.py", line 10614, in _check_publish_status
    raise Exception("Job failed.")
Exception: Job failed.

----------------

# arcgis-1.9.0
...
  File "~\agol_overwrite.py", line 240, in overwrite_agol_biweekly
    flcollection.manager.overwrite(csvowriteversion_path)
  File "~\venv\lib\site-packages\arcgis\features\managers.py", line 2036, in overwrite
    if bool(self._fs.replicas.get_list()):
AttributeError: 'FeatureLayerCollection' object has no attribute 'replicas'

Traceback (most recent call last):
  File "C:\Users\Alex.Cooper\Documents_Local\coding_loc\local_ssm\py_venv\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3553, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-15-e4b5a6775c63>", line 1, in <module>
    xstcsv.publish(overwrite=True)
  File "C:\Users\Alex.Cooper\Documents_Local\coding_loc\local_ssm\py_venv\venv\lib\site-packages\arcgis\gis\__init__.py", line 10387, in publish
    serviceitem_id = self._check_publish_status(ret, folder)
  File "C:\Users\Alex.Cooper\Documents_Local\coding_loc\local_ssm\py_venv\venv\lib\site-packages\arcgis\gis\__init__.py", line 10614, in _check_publish_status
    raise Exception("Job failed.")
Exception: Job failed.

[JakeSkinner]

Hi @JACooper, I've seen issues where feature services been somewhat 'corrupted'.  As a test, try publishing the data as a new service, and try overwriting the new service to see if that works.  If it works, something is most likely corrupt with the service.

[JACooper]

@JakeSkinner  I've run into this occasionally, but it's very rare. I published a few different services from the same and slightly altered CSVs, and each one is loaded and is able to be overwritten without issue. Same schema as the apparently corrupt service.

Is there a known reason as to why feature services become corrupt? I can't think of anything unusual that has happened lately to the service in question that would lead to this occurring. And are there known fixes to these corruptions that don't involve a tear-down, reload solution?

[JACooper]

I am wondering if you're right and this is the lurking background issue: https://community.esri.com/t5/arcgis-online-questions/hosted-feature-layer-view-does-not-update-when.... Still testing to see if this is the case, but I have also lost the ability to configure my downstream hosted feature layer view, similar to this issue.

[timcneil]

@JACooper When you try to Overwrite the service using the Update Data UI are you seeing any console errors or failing requests in the network traffic? 

[JACooper]

@timcneil good thought. I got a couple different errors. All pasted below.

The first was when I tried to overwrite from the web UI.

The second was when the overwrite had fully failed and I was on the https://ccaoa.maps.arcgis.com/home/item.html?id=redacted&jobid=thejobid page. I tried to overwrite again and got the `cannot read properties of undefined "sourceSchemaChangesAllowed"` error. Googling the latter didn't help me make sense of it, but perhaps this gives you a hint at what is happening here?

{
    "status": "failed",
    "statusMessage": "failed",
    "itemId": "{redacted}",
    "_ssl": "undefined"
}

[KentuckyDGI]

I had issues also.  My log files said it had to do with delete protection, but it wasn't the issue.  I found that after much frustration, that there was an error in the metadata in ArcGIS Online.  I had to add country origin language.  My process ran flawlessly for 3 years until November 13. 

I gained this knowledge by chance when a tech told me that she had been seeing issues related to metadata and the most recent update.  When we got off the phone I checked my metadata and there was an error.

It's mind-baffling.  I fixed my metadata, and my overwrite process works again. 

 

 

[JACooper]

@KentuckyDGI I appreciate the suggestion, and the symptoms look very close! However, my organization does not enable metadata. I don't have that view at all on my item's properties page. So I don't think this could be the situation for, could it? I dug around in the layer's properties in the python API and didn't see anything like this to set there.

[KentuckyDGI]

Gosh.  I'm sorry to hear that.  All I can tell you is that at one point in the distant past, we had a "broken" issue on several layers, and long story short, it was a metadata releated issue.  I am not saying your issue is because of metadata, since you said that is disabled.  But honestly, I wouldn't discount the relation. 

Like you, I was very hesistant to make any changes because the default recommendation "delete and republish" is not an acceptable solution.  Your data is childcare, mine is 911 related.  Both very critical operations and like you, I was just not comfortable doing something so drastic as deleting and republishing.

This might not be within in your scope, but what if your organization enabled metadata, then you reviewed the metadata in AGOL interface and either deleted all metadata or made suggested corrections.  Again, apologies if that's out of your hands.

Bottom line, this is an issue created by the November update, and ESRI needs to fix it and give us explanations.

Best of luck.  Let me know how it goes.

[KentuckyDGI]

In different words, and some point, metadata may have been introduced into the mix, and possibly even removed, but persisted in AGOL (regardless of the enablement).  This update hosed that whole process