topic Re: Append selected records and calculate value for one field in ArcGIS API for Python Questions

Append selected records and calculate value for one field

AmyRoust — Tue, 12 Nov 2024 16:39:59 GMT

I'm new to using the ArcGIS API for Python in an ArcGIS Online Notebook and am looking for help to identify the right function(s) for a data maintenance script I'm migrating from a scheduled task on a server.

The start of the script works fine: I select a small number of records from a hosted feature layer. (If it matters, the hosted feature layer is part of a Survey123 survey).

I'm struggling with finding the function(s) to:

1. Append those selected records to another hosted feature layer, and

2. On those newly appended records, update the value in the status field to 'Not Reviewed'. The status field does not exist in the source layer, but it does in the target layer.

These two steps don't have to be in this order. I'm fine with exporting the records, adding a status field, calculating that 'Not Reviewed' value and then appending the updated records to my target layer.

Appreciate any hints that the community can offer!

Re: Append selected records and calculate value for one field

JakeSkinner — Wed, 13 Nov 2024 12:23:55 GMT

Hi @AmyRoust,

You could do something like below:

from arcgis.gis import GIS # Connect to your GIS gis = GIS("https://www.arcgis.com", "username", "password") # Access source feature service item = gis.content.get("source item id") feature_layer = item.layers[0] # Define your query query = "POPULATION > 10000" # Query the feature layer and create a spatial dataframe sdf = feature_layer.query(where=query).sdf # Calculate the status field sdf["status field name"] = sdf.calculate("'Not Reviewed'", inplace=True) # Access the hosted feature service to update item = gis.content.get("target item id") feature_layer = item.layers[0] # Append the data from the spatial dataframe to the hosted feature service added_features = feature_layer.edit_features(adds=sdf.to_featureset())

Re: Append selected records and calculate value for one field

Clubdebambos — Wed, 13 Nov 2024 08:13:06 GMT

Hi @JakeSkinner,

Just an observation... in line 14 you convert the feature set (returned from the query) to a sdf, and then in line 21 you are converting the sdf back to a feature set with no operations on the sdf in-between.

Would the operation in-between be to add the "status" field to the sdf, populate the status attributes to Not Reviewed and then append into the target? (omitting the calculate method on line 24)

@AmyRoust do you also need to account for attachments?

Cheers,

Glen

Re: Append selected records and calculate value for one field

JakeSkinner — Wed, 13 Nov 2024 12:24:37 GMT

Yes, you are correct @Clubdebambos, thanks for pointing that out. I went ahead and updated my previous reply.

Re: Append selected records and calculate value for one field

AmyRoust — Wed, 13 Nov 2024 15:25:29 GMT

I don't have any attachments, @Clubdebambos.

Is there a way to do field mapping with the edit_features function? Some of the field names in the input feature layer don't match the target layer.

Re: Append selected records and calculate value for one field

JakeSkinner — Wed, 13 Nov 2024 17:58:53 GMT

@AmyRoust try the following:

# Define the field mapping between the source and target dataframes field_mapping = {'ID': 'ID_new', 'Name': 'Name_new', 'Value': 'Value_new'} # Append data with field mapping feature_layer.edit_features(adds=sdf.to_featureset(), field_mapping=field_mapping)

Re: Append selected records and calculate value for one field

AmyRoust — Tue, 19 Nov 2024 22:26:13 GMT

I still have something wrong, but I don't know how to print error messages because arcpy.GetMessages() doesn't return anything. Is there another way to get the error messages from failed functions?

Here's the portion of code that is failing:

field_mapping = {'ObservationDate':'ReportObservationDate', 'CampsiteDescription':'ReportCampsiteDescription', 'FollowUpRequest':'ReportFollowUpRequest', 'SubmitterName':'ReportSubmitterName', 'PreferredContact':'ReportSubmitterPreferredContact', 'EmailAddress':'ReportSubmitterEmailAddress', 'PhoneNumber':'ReportSubmitterPhoneNumber', 'submission_date_and_time':'ReportDate' } sdf = feature_layer.query(where=query).sdf feature_layer.edit_features(adds = sdf.to_featureset(), field_mapping=field_mapping)

I think line 10 is working because I added a print statement for the sdf variable and got this (confidential information blurred in the screenshot):

Re: Append selected records and calculate value for one field

AmyRoust — Tue, 19 Nov 2024 22:28:25 GMT

Just for clarification, I do have the variable feature_layer defined earlier in the script and it's working properly.

item = gis.content.get("999999999") # actual item id has been removed for this chat feature_layer = item.layers[0]

Re: Append selected records and calculate value for one field

JakeSkinner — Wed, 20 Nov 2024 12:43:51 GMT

Try the following:

try: feature_layer.edit_features(adds = sdf.to_featureset(), field_mapping=field_mapping) except Exception as e: print(e) arcpy.AddError(str(e))

Re: Append selected records and calculate value for one field

AmyRoust — Thu, 21 Nov 2024 16:25:18 GMT

Thank you! That worked for error trapping. I fixed the issue, but field mapping doesn't work.

if len(selected_features.features)>0: field_mapping = {'ObservationDate':'ReportObservationDate', 'CampsiteDescription':'ReportCampsiteDescription', 'FollowUpRequest':'ReportFollowUpRequest', 'SubmitterName':'ReportSubmitterName', 'PreferredContact':'ReportSubmitterPreferredContact', 'EmailAddress':'ReportSubmitterEmailAddress', 'PhoneNumber':'ReportSubmitterPhoneNumber', 'submission_date_and_time':'ReportDate' } sdf = feature_layer.query(where=query) feature_layer.edit_features(adds = sdf, field_mapping=field_mapping)

Line 12 generates the error:

FeatureLayer.edit_features() got an unexpected keyword argument 'field_mapping'
ERROR: FeatureLayer.edit_features() got an unexpected keyword argument 'field_mapping'

I tried using field_mappings (plural) since that's listed in the FeatureLayer append method as a keyword, but no luck.

So then I thought I'd try the append method, but I'm getting Unknown Error (Error Code: 500) in response to this:

if len(selected_features.features)>0: field_mapping = [{'ObservationDate':'ReportObservationDate', 'CampsiteDescription':'ReportCampsiteDescription', 'FollowUpRequest':'ReportFollowUpRequest', 'SubmitterName':'ReportSubmitterName', 'PreferredContact':'ReportSubmitterPreferredContact', 'EmailAddress':'ReportSubmitterEmailAddress', 'PhoneNumber':'ReportSubmitterPhoneNumber', 'submission_date_and_time':'ReportDate' }] sdf = feature_layer.query(where=query) feature_layer.append(sdf, field_mappings = field_mapping, return_messages = True)

My guess is that the error code is the result of not specifying an item_id, since sdf is a selection of records and not an entire item.

Any ideas on what to try next?

I wish the ArcGIS API for Python documentation consistently provided sample code like the arcpy documentation does.

Re: Append selected records and calculate value for one field

AmyRoust — Wed, 04 Dec 2024 21:20:11 GMT

I'm a lot closer now after making quite a few changes. Now I'm getting the error message:

A general error occurred: Circular reference detected
ERROR: A general error occurred: Circular reference detected

I believe that the error is in line 42. I replaced some info with ... for security purposes.

from arcgis.gis import GIS gis = GIS("home") # Import arcpy module from arcgis.gis import GIS from arcgis.features import FeatureLayer, GeoAccessor # might not need that last one? import time, arcpy, os, datetime # Variables s123Layer = r'https://services.arcgis.com/.../FeatureServer/0' report_copy_url = r'https://services.arcgis.com/.../FeatureServer/0' try: item = gis.content.get("...") feature_layer = item.layers[0] field_name = "ReportDate" query_result = feature_layer.query(out_statistics=[{"statisticType": "max", "onStatisticField": field_name}]) milliseconds = query_result.features[0].attributes[f"MAX_{field_name}"] date_time = datetime.datetime.utcfromtimestamp(milliseconds / 1000.0) max_date = date_time.strftime('%Y-%m-%d %H:%M:%S') print("Most recent report date in ...:", max_date) feature_layer = FeatureLayer(s123Layer, gis=gis) query = f"submission_date_and_time > timestamp '{max_date}'" selected_features = feature_layer.query(where=query) print(f"Number of new features to append to ...: {len(selected_features)}") ###### SCRIPT WORKS TO THIS POINT ####### if len(selected_features.features)>0: fm = [{'ObservationDate':'ReportObservationDate', 'CampsiteDescription':'ReportCampsiteDescription', 'FollowUpRequest':'ReportFollowUpRequest', 'SubmitterName':'ReportSubmitterName', 'PreferredContact':'ReportSubmitterPreferredContact', 'EmailAddress':'ReportSubmitterEmailAddress', 'PhoneNumber':'ReportSubmitterPhoneNumber', 'submission_date_and_time':'ReportDate' }] targetLayer = FeatureLayer(report_copy_url,gis=gis) targetLayer.append(selected_features.features, field_mappings = fm, return_messages = True) else: print("No new reports to load into ...") except Exception as e: print(e) arcpy.AddError(str(e))

@JakeSkinner @Clubdebambos - do you see any issues?

Re: Append selected records and calculate value for one field

JakeSkinner — Tue, 10 Dec 2024 13:52:25 GMT

@AmyRoust I see your importing arcpy. Are you using an Advanced runtime in AGOL Notebooks?

Re: Append selected records and calculate value for one field

AmyRoust — Wed, 11 Dec 2024 14:29:48 GMT

Good question! I believe I am:

Re: Append selected records and calculate value for one field

JakeSkinner — Mon, 16 Dec 2024 12:05:19 GMT

@AmyRoust since you are using arcpy, I would use these tools to perform the append. You can perform these steps in ArcGIS Pro and copy the code to your Notebook. For example, use the Make Feature Layer tool to create a feature layer with the query. Then, use the Append tool to append the records with the field mapping configured. You can easily copy the code you need by clicking the dropdown arrow next to Run: