nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Append selected records and calculate value for one field

AmyRoust

I'm new to using the ArcGIS API for Python in an ArcGIS Online Notebook and am looking for help to identify the right function(s) for a data maintenance script I'm migrating from a scheduled task on a server.

The start of the script works fine: I select a small number of records from a hosted feature layer. (If it matters, the hosted feature layer is part of a Survey123 survey).

I'm struggling with finding the function(s) to:

1. Append those selected records to another hosted feature layer, and

2. On those newly appended records, update the value in the status field to 'Not Reviewed'. The status field does not exist in the source layer, but it does in the target layer.

These two steps don't have to be in this order. I'm fine with exporting the records, adding a status field, calculating that 'Not Reviewed' value and then appending the updated records to my target layer.

Appreciate any hints that the community can offer!

Vind meer berichten getagd met

Accepted answers

All comments

JakeSkinner

@AmyRoust since you are using arcpy, I would use these tools to perform the append. You can perform these steps in ArcGIS Pro and copy the code to your Notebook. For example, use the Make Feature Layer tool to create a feature layer with the query. Then, use the Append tool to append the records with the field mapping configured. You can easily copy the code you need by clicking the dropdown arrow next to Run:

AmyRoust

Good question! I believe I am:

JakeSkinner

@AmyRoust I see your importing arcpy. Are you using an Advanced runtime in AGOL Notebooks?

AmyRoust

I'm a lot closer now after making quite a few changes. Now I'm getting the error message:

A general error occurred: Circular reference detected
ERROR: A general error occurred: Circular reference detected

I believe that the error is in line 42. I replaced some info with ... for security purposes.

from arcgis.gis import GIS
gis = GIS("home")

# Import arcpy module
from arcgis.gis import GIS
from arcgis.features import FeatureLayer, GeoAccessor # might not need that last one?
import time, arcpy, os, datetime

# Variables
s123Layer = r'https://services.arcgis.com/.../FeatureServer/0'
report_copy_url = r'https://services.arcgis.com/.../FeatureServer/0'

try:
    item = gis.content.get("...")
    feature_layer = item.layers[0]
    field_name = "ReportDate"
    query_result = feature_layer.query(out_statistics=[{"statisticType": "max", "onStatisticField": field_name}])
    milliseconds = query_result.features[0].attributes[f"MAX_{field_name}"]
    date_time = datetime.datetime.utcfromtimestamp(milliseconds / 1000.0)
    max_date = date_time.strftime('%Y-%m-%d %H:%M:%S')
    print("Most recent report date in ...:", max_date)

    feature_layer = FeatureLayer(s123Layer, gis=gis)
    query = f"submission_date_and_time > timestamp '{max_date}'"
    selected_features = feature_layer.query(where=query)
    print(f"Number of new features to append to ...: {len(selected_features)}")
    
###### SCRIPT WORKS TO THIS POINT #######
    
    if len(selected_features.features)>0:
        fm = [{'ObservationDate':'ReportObservationDate',
                         'CampsiteDescription':'ReportCampsiteDescription',
                         'FollowUpRequest':'ReportFollowUpRequest',
                         'SubmitterName':'ReportSubmitterName',
                         'PreferredContact':'ReportSubmitterPreferredContact',
                         'EmailAddress':'ReportSubmitterEmailAddress',
                         'PhoneNumber':'ReportSubmitterPhoneNumber',
                         'submission_date_and_time':'ReportDate'
                        }]
        
        targetLayer = FeatureLayer(report_copy_url,gis=gis)
        targetLayer.append(selected_features.features, field_mappings = fm, return_messages = True)

    else:
        print("No new reports to load into ...")

except Exception as e:
    print(e)
    arcpy.AddError(str(e))

@JakeSkinner @Clubdebambos - do you see any issues?

AmyRoust

Thank you! That worked for error trapping. I fixed the issue, but field mapping doesn't work.

    if len(selected_features.features)>0:
        field_mapping = {'ObservationDate':'ReportObservationDate',
                         'CampsiteDescription':'ReportCampsiteDescription',
                         'FollowUpRequest':'ReportFollowUpRequest',
                         'SubmitterName':'ReportSubmitterName',
                         'PreferredContact':'ReportSubmitterPreferredContact',
                         'EmailAddress':'ReportSubmitterEmailAddress',
                         'PhoneNumber':'ReportSubmitterPhoneNumber',
                         'submission_date_and_time':'ReportDate'
                        }
        sdf = feature_layer.query(where=query)
        feature_layer.edit_features(adds = sdf, field_mapping=field_mapping)

Line 12 generates the error:

FeatureLayer.edit_features() got an unexpected keyword argument 'field_mapping'
ERROR: FeatureLayer.edit_features() got an unexpected keyword argument 'field_mapping'

I tried using field_mappings (plural) since that's listed in the FeatureLayer append method as a keyword, but no luck.

So then I thought I'd try the append method, but I'm getting Unknown Error (Error Code: 500) in response to this:

    if len(selected_features.features)>0:
        field_mapping = [{'ObservationDate':'ReportObservationDate',
                         'CampsiteDescription':'ReportCampsiteDescription',
                         'FollowUpRequest':'ReportFollowUpRequest',
                         'SubmitterName':'ReportSubmitterName',
                         'PreferredContact':'ReportSubmitterPreferredContact',
                         'EmailAddress':'ReportSubmitterEmailAddress',
                         'PhoneNumber':'ReportSubmitterPhoneNumber',
                         'submission_date_and_time':'ReportDate'
                        }]
        sdf = feature_layer.query(where=query)
        feature_layer.append(sdf,
                            field_mappings = field_mapping,
                            return_messages = True)

My guess is that the error code is the result of not specifying an item_id, since sdf is a selection of records and not an entire item.

Any ideas on what to try next?

I wish the ArcGIS API for Python documentation consistently provided sample code like the arcpy documentation does.

JakeSkinner

Try the following:

try:
    feature_layer.edit_features(adds = sdf.to_featureset(), field_mapping=field_mapping)
except Exception as e:
    print(e)
    arcpy.AddError(str(e))

AmyRoust

Just for clarification, I do have the variable feature_layer defined earlier in the script and it's working properly.

item = gis.content.get("999999999") # actual item id has been removed for this chat
feature_layer = item.layers[0]

AmyRoust

I still have something wrong, but I don't know how to print error messages because arcpy.GetMessages() doesn't return anything. Is there another way to get the error messages from failed functions?

Here's the portion of code that is failing:

field_mapping = {'ObservationDate':'ReportObservationDate',
                         'CampsiteDescription':'ReportCampsiteDescription',
                         'FollowUpRequest':'ReportFollowUpRequest',
                         'SubmitterName':'ReportSubmitterName',
                         'PreferredContact':'ReportSubmitterPreferredContact',
                         'EmailAddress':'ReportSubmitterEmailAddress',
                         'PhoneNumber':'ReportSubmitterPhoneNumber',
                         'submission_date_and_time':'ReportDate'
                        }
        sdf = feature_layer.query(where=query).sdf
        feature_layer.edit_features(adds = sdf.to_featureset(), field_mapping=field_mapping)

I think line 10 is working because I added a print statement for the sdf variable and got this (confidential information blurred in the screenshot):

JakeSkinner

@AmyRoust try the following:

# Define the field mapping between the source and target dataframes
field_mapping = {'ID': 'ID_new', 'Name': 'Name_new', 'Value': 'Value_new'}

# Append data with field mapping
feature_layer.edit_features(adds=sdf.to_featureset(), field_mapping=field_mapping)

AmyRoust

I don't have any attachments, @Clubdebambos.

Is there a way to do field mapping with the edit_features function? Some of the field names in the input feature layer don't match the target layer.

JakeSkinner

Yes, you are correct @Clubdebambos, thanks for pointing that out. I went ahead and updated my previous reply.

Clubdebambos

Hi @JakeSkinner,

Just an observation... in line 14 you convert the feature set (returned from the query) to a sdf, and then in line 21 you are converting the sdf back to a feature set with no operations on the sdf in-between.

Would the operation in-between be to add the "status" field to the sdf, populate the status attributes to Not Reviewed and then append into the target? (omitting the calculate method on line 24)

@AmyRoust do you also need to account for attachments?

Cheers,

Glen

JakeSkinner

Hi @AmyRoust,

You could do something like below:

from arcgis.gis import GIS

# Connect to your GIS
gis = GIS("https://www.arcgis.com", "username", "password")

# Access source feature service
item = gis.content.get("source item id")
feature_layer = item.layers[0]

# Define your query
query = "POPULATION > 10000"

# Query the feature layer and create a spatial dataframe
sdf = feature_layer.query(where=query).sdf

# Calculate the status field
sdf["status field name"] = sdf.calculate("'Not Reviewed'", inplace=True)

# Access the hosted feature service to update
item = gis.content.get("target item id")
feature_layer = item.layers[0]

# Append the data from the spatial dataframe to the hosted feature service
added_features = feature_layer.edit_features(adds=sdf.to_featureset())

Doe mee aan het gesprek

Aangemelde leden kunnen berichten plaatsen, updates volgen en meer. Nieuw hier? Registreer een gratis account.

Snelle Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.