Unknown error when appending feature layer in arcpy

KitTompkins · ‎02-09-2023

After failing to manage this in Model Builder, I have been teaching myself python for almost a month so that I can automate updates to my organization's web layers. (There are about 50, so its a real drag to do this by hand whenever I make edits.)

I have code that I think should work, and PyCharm detects no errors, but when I run it, it gets right to the append part and throws Exception: Unknown Error.

import arcpy
import os
import uuid
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection
from zipfile import ZipFile
import datetime

# Overwrite Output
arcpy.env.overwriteOutput = True

# Create UUID variable for GDB
gdbId = str(uuid.uuid1())

print("Connecting to AGOL....")

sourceURL = 'https://someorg.maps.arcgis.com/'
sourceAdmin = "somebody"
sourcePassword = "apassword"
source = GIS(sourceURL, sourceAdmin, sourcePassword)
me = source.users.me
workspace = r'D:\Master_GIS_Files\Data\somedata\data.gdb'
fgdb = r'D:\Master_GIS_Files\Data\somedata\data/'

print("Creating temporary File Geodatabase")
gdb = arcpy.CreateFileGDB_management(arcpy.env.scratchFolder, gdbId)[0]
temp = gdb + ".zip"


def zipDir(dirPath, zipPath):
    zipf = ZipFile(zipPath, mode='w')
    tgdb = os.path.basename(dirPath)
    for root, _, files in os.walk(dirPath):
        for file in files:
            if 'lock' not in file:
                filePath = os.path.join(root, file)
                zipf.write(filePath, os.path.join(tgdb, file))
    zipf.close()


print("Zipping temp FGD")
zipDir(fgdb, temp)

print("Uploading File Geodatabase")
fgd_properties = {'title': gdbId, 'tags': 'temp file geodatabase', 'type': 'File Geodatabase'}
fgd_item = source.content.add(item_properties=fgd_properties, data=temp)
nID = fgd_item.id


stat = 'yes'

def StuffManager(q):
    Lyr = ''
    def FCIterator():
        with arcpy.EnvManager(workspace=workspace):
            gd = arcpy.Describe(workspace)
            fds = gd.children
            for fd in fds:
                fdd = fd.name
                fds = arcpy.Describe(fdd)
                fcs = fds.children
                for fcc in fcs:
                    fc = fcc.name
                    print(f'Loading feature class {fc}')
                    yield fc

    def AppendWebLayer(lyr, lID):
        svc = source.content.get(lID)
        serviceLayer = FeatureLayerCollection.fromitem(svc)
        flyr = serviceLayer.layers[0]
        updateTime = datetime.datetime.fromtimestamp(svc.modified / 1000)
        new_field = {"name": "Owner","type": "esriFieldTypeString","alias": "Owner","length": 50,"nullable": True,"editable": True,"visible": True,"domain": None}
        update_dict = {"fields": [new_field]}
        if updateTime != datetime.date.today():
            flyr.manager.add_to_definition(update_dict)
            flyr.manager.truncate()
            flyr.append(item_id=nID, upload_format='filegdb', source_table_name=lyr, upsert=True, skip_updates=False, update_geometry=True, skip_inserts=False, upsert_matching_field='OBJECTID', return_messages=True)
            return lyr, 'yes'
        else:
            return lyr, 'no'

    def GetWebLayer(layer):
        items = me.items(folder='Utilities')
        ftrlyrs = [i for i in items if i.title == layer]
        for f in ftrlyrs:
            if f.type == "Feature Service":
                flID = f.id
                title = f.title
                print(f'Found web layer {title} with ID {flID}.')
                return AppendWebLayer(layer, flID)


    while q == 'yes':
        ftr = FCIterator()
        values = GetWebLayer(next(ftr))
        Lyr = values[0]
        q = values[1]
        print(f"Layer {Lyr} updated!")
    else:
        print(f"Layer {Lyr} has already been updated.")
StuffManager(stat)
print('Deleting geodatabase from AGOL')
thing = source.content.get(nID)
thing.delete()
print('Script complete')

Here is the error message:

File "D:\Programs\ArcGISPro\ArcGISPro\bin\Python\envs\arcgispro-py3\lib\site-packages\IPython\core\interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-7ddf5671b1b3>", line 1, in <module>
runfile('C:\\Users\\ThisUser\\PycharmProjects\\automateupdateweblayer\\Iterate and append', wdir='C:\\Users\\ThisUser\\PycharmProjects\\automateupdateweblayer')
File "C:\ThisUser\AppData\Local\JetBrains\Toolbox\apps\PyCharm-E\ch-0\222.4345.35\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "C:\Users\ThisUser\AppData\Local\JetBrains\Toolbox\apps\PyCharm-E\ch-0\222.4345.35\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:\Users\ThisUser\PycharmProjects\automateupdateweblayer\Iterate and append", line 101, in <module>
StuffManager(stat)
File "C:\Users\ThisUser\PycharmProjects\automateupdateweblayer\Iterate and append", line 95, in ShitManager
values = GetWebLayer(next(ftr))
File "C:\Users\ThisUser\PycharmProjects\automateupdateweblayer\Iterate and append", line 90, in GetWebLayer
return AppendWebLayer(layer, flID)
File "C:\Users\ThisUser\PycharmProjects\automateupdateweblayer\Iterate and append", line 77, in AppendWebLayer
flyr.append(item_id=nID, upload_format='filegdb', source_table_name=lyr, upsert=True, skip_updates=False, update_geometry=True, skip_inserts=False, upsert_matching_field='OBJECTID', return_messages=True)
File "D:\Programs\ArcGISPro\ArcGISPro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\layer.py", line 2384, in append
return self._check_append_status(res, return_messages)
File "D:\Programs\ArcGISPro\ArcGISPro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\layer.py", line 2393, in _check_append_status
sres = self._con.get(path=surl, params={"f": "json"})
File "D:\Programs\ArcGISPro\ArcGISPro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 763, in get
return self._handle_response(
File "D:\Programs\ArcGISPro\ArcGISPro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 900, in _handle_response
self._handle_json_error(data["error"], errorcode)
File "D:\Programs\ArcGISPro\ArcGISPro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 923, in _handle_json_error
raise Exception(errormessage)
Exception: Unknown Error
(Error Code: 500)

If anybody could help me figure out what the heck is wrong with this stupid code, I'd be grateful.

Anonymous User · ‎02-09-2023

Wrap your processes up in try/ excepts and try to catch the error. For example:

def AppendWebLayer(lyr, lID):
    try:
        svc = source.content.get(lID)
        serviceLayer = FeatureLayerCollection.fromitem(svc)
        flyr = serviceLayer.layers[0]
        updateTime = datetime.datetime.fromtimestamp(svc.modified / 1000)
        new_field = {"name": "Owner", "type": "esriFieldTypeString", "alias": "Owner", "length": 50, "nullable": True,
                     "editable": True, "visible": True, "domain": None}
        update_dict = {"fields": [new_field]}
        if updateTime != datetime.date.today():
            flyr.manager.add_to_definition(update_dict)
            flyr.manager.truncate()
            flyr.append(item_id=nID, upload_format='filegdb', source_table_name=lyr, upsert=True, skip_updates=False,
                        update_geometry=True, skip_inserts=False, upsert_matching_field='OBJECTID', return_messages=True)
            return lyr, 'yes'
        else:
            return lyr, 'no'
    except Exception as ex:
        arcpy.AddMessage(ex)

This might sound odd, but for two of my scripts that involve the arcgis module, I have to run them in debug mode for them to work and they run without exceptions. When I run them as production, exceptions are thrown. Might try running it in debug mode.

Kepa · ‎02-10-2023

Hello @KitTompkins,

Looking at the traceback it seems some kind of issue with the connection when trying to request _check_append_status. Code 500 stands for Internal Server Error (quite broad error related to the target server, in this case your organization).

I'd suggest trying to append it manually and check if it crashes. If not, I'd execute the code line by line on the console (at least the AppendWebLayer function). Sometimes following the workflow just helps to solve the problem. Also check if the target service has the Append capability enabled if you are running the script with a nonadministrative user.

And like @Anonymous User said, it is strongly recommended to use try/except blocks to catch any error, although Exception is too general class is enough when you don't know what kind of exception may arise.

Regards,

KitTompkins · ‎02-13-2023

When I used the try: except and got messages it said the layer didn't exist, even though it found the web layer in the previous step. I'm wondering if using truncate changes the id or something.

Kepa · ‎02-13-2023

In line 77 you are passing the following paramater:

source_table_name=lyr

But lyr on the code is the layer you get from GetWebLayer (layers from AGOL's folder). That parameter must be the name of the layer in the geodatabase you just uploaded. Unless they're called completely equal it's going to output an error 500. I managed to reproduce your error passing an unexisting layer in that parameter:

Hope that helps!