topic Pull Attachments Table from AGOL in Data Management Questions

Pull Attachments Table from AGOL

CodyPatterson — Wed, 31 Jul 2024 13:08:52 GMT

Hey All,

I have recently been asked to create a sync between AGOL and a local database, this is a relatively simple task with a combination of ArcGIS API for Python and ArcPy since ArcPy cannot communicate with AGOL.

The issue arises now because there are attachments, when I exported the feature layer to a FGDB I see this here:

I see that there is a relationship and an ATTACH table, I found that the relationship can be entered once and left without issue, but I need to update the ATTACH table for the attachments in the feature classes to be included, how can I extract this ATTACH table or possibly the attachments to format into the table?

Cody

Re: Pull Attachments Table from AGOL

RhettZufelt — Wed, 31 Jul 2024 15:12:55 GMT

To sync from AGOL data to the local database, I have been using the arcpy.management.Append with preserve GlobalIds and maintain Attachments.

Then I get an updated local copy of the data with attachments while preserving the GlobalIds.

Re: Pull Attachments Table from AGOL

CodyPatterson — Thu, 01 Aug 2024 11:39:35 GMT

Hey @RhettZufelt

Unfortunately it seems that ArcPy is not reliable with AGOL according to BUG-000144037 which is not planned to be fixed unfortunately, I ended up setting up a macro on a server to make clicks which downloaded the FGDB, and then ran the script after it was export from AGOL, which then is deleted afterward.

Cody

Re: Pull Attachments Table from AGOL

RhettZufelt — Mon, 05 Aug 2024 15:24:51 GMT

Not sure if you are trying to sync both ways, or why you are wanting to use the arcpy.MakeFeatureLayer_management as the bug references, but maybe my process is different enough it won't work for you.

In my case, I have a local eGDB that I published to AGOL for data collection in a Field Maps app.

So, the data is collected in AGOL hosted feature service, but I need to keep the local copy up to date.

I have been using code similar to the following for the last 2 years and have had no issues with it.

Basically, I establish a search cursor on the AGOL and local datasets and load the GlobalIDs into a list. It also compares the two lists, and if there are GlobalIDs in the AGOL list, but not the local list, it means those features need to be copied (appended) over. (So, we are only coping the 'missing' features, not the entire dataset)

Then I use arcpy.management.Append to append the missing features to the local data using a where clause that is an IN() with the GlobalID's from the missing list like so:

import arcpy Signs = r'\\path_to\\SignDatabase.sde\SignsDatabase.DBO.Signs' # path to the signs local eGDB FC L0Sign = r"https://services3.arcgis.com/itemID/ArcGIS/rest/services/HostedFeatureService/FeatureServer/0" # URL to AGOL hosted feature AGOL_SignTable = arcpy.FeatureSet() # define AGOL_SignTable as FeatureSet AGOL_SignTable.load(L0Sign) # Load hosted feature into AGOL_SignTable SignList = [] # empty list for local GDB features AGOLSignList = [] # empty list for AGOL features with arcpy.da.SearchCursor(Signs,"GlobalID") as cursor: # Get list of GlobalID's from SDE data for row in cursor: SignList.append(row[0]) with arcpy.da.SearchCursor(AGOL_SignTable,"GlobalID") as cursor: # Get list of GlobalID's from AGOL data for row in cursor: AGOLSignList.append(row[0].upper()) if (row[0].upper() or row[0]) not in SignList: # and if not in local SignList, append to AddSignList AddSignList.append(row[0]) ith arcpy.EnvManager(preserveGlobalIds=True, maintainAttachments = True): if AddSignList: # if any 'missing' features in AddSignList, then append to local eGDB print("adding " + str(len(AddSignList)) + " Sign Records") Signs = arcpy.management.Append(inputs=[L0Sign], target=Signs, schema_type="TEST", field_mapping="", subtype="", expression="GlobalID IN ('" + '\',\''.join(AddSignList) + "')")[0] else: print("No Signs to add")

I can't say the same for any scripts I run using the API python (seem to get a lot of timeout errors, etc. and have to code accordingly). However, I have had good success with the API code here to export an AGOL hosted feature service to a FGDB (including attachments) and download to local folder without having to script any mouse clicks.

Re: Pull Attachments Table from AGOL

CodyPatterson — Fri, 02 Aug 2024 11:22:43 GMT

Hey @RhettZufelt

Thank you for the informative response and code, I attempted to copy the code exactly as you mentioned, but I'm not sure what the SignTable function should represent. The reason behind the makefeaturelayer function is due to geoprocessing tools that need to run on the hosted feature service. Attempting to load the layer into an FGDB does not work as well, currently, the main issue is getting ArcPy and AGOL to talk in any extent, the support technician I worked with stated that it's a known limitation that these aren't working, but work for some just fine, and ESRI is not interesting in pursuing the fix further, as the bug mentions unfortunately.

Re: Pull Attachments Table from AGOL

RhettZufelt — Mon, 05 Aug 2024 19:42:26 GMT

If you copied it exactly, not surprising you got an error as I had a mistake in the code.

AGOL_SignTable = arcpy.FeatureSet() AGOL_SignTable.load(L0Sign)

Basically, I define AGOL_SignTable as an arcpy feature set, then the next line loads the data from the hosted feature server layer URL into the arcpy FeatureSet.

Once loaded, I can treat AGOL_SignTable as any other FGDB featureclass. I have not tested all functions on it, but works well with search cursors, arcpy calculate and arcpy append.

Re: Pull Attachments Table from AGOL

CodyPatterson — Mon, 05 Aug 2024 18:17:39 GMT

Hey @RhettZufelt

This is my output here:

My code:

import arcpy layer_name = r"https://services9.arcgis.com/{id}/arcgis/rest/services/USA_Counties_6260724028191129561/FeatureServer" AGOL_SignTable = arcpy.FeatureSet() AGOL_SignTable.load(layer_name)

This was another error that was mentioned to only work for some, and not others.

Cody

Re: Pull Attachments Table from AGOL

RhettZufelt — Mon, 05 Aug 2024 19:44:48 GMT

What happens is you use the full path to the feature layer in that feature service (add the correct layer number to the end of the URL. /0 on my HFS)?

layer_name = r"https://services9.arcgis.com/{id}/arcgis/rest/services/USA_Counties_6260724028191129561/FeatureServer/0"

Re: Pull Attachments Table from AGOL

CodyPatterson — Wed, 07 Aug 2024 11:18:28 GMT

Hey @RhettZufelt

Unfortunately, the exact same error came through. Not sure why my AGOL and ArcPy are so incompatible, but maybe I'm just missing something!

Cody

Re: Pull Attachments Table from AGOL

RhettZufelt — Wed, 07 Aug 2024 14:56:40 GMT

Not sure what to tell you. I have tested on my end with polygon and point feature classes with no issue.

Perhaps there is a limitation on the Feature Service you are connecting to. I published my own HFS with no restrictions and use the code to keep the local data in sync with changes to the HFS.