Help with comparing values in 1 feature service against features in second feature service

Question

Here's what I'm trying to do. I have a feature service (FS1) which contains inspections. These inspections can be identified by values in the following fields: plantation, years_planted, date_of_inspection & inspection_number.

Feature service 2 (FS2) contains 1 row per plantation and planting year (e.g. Ferndale:2024, Ferndale:2025, etc.) and a number of fields to record if an inspection has been completed or not based on a regular cycle.

Below is a list of values from FS1's inspection_number field and their corresponding field name from FS2.

'2 weeks': 'insp_2_weeks'
'4 weeks': 'insp_4_weeks'
'6 weeks': 'insp_6_weeks'
'8 weeks': 'insp_8_weeks'

When an inspection is done in FS1, the corresponding row in FS2 should be updated to record that it has happened by updating the corresponding inspection_number field with "username on inspection_date".

Below is the code I have so far. I know it works up to Line 80 based on print statements returned. I can't seem to get it to match up the features from FS1 (lots of inspections) with FS2 (inspection tracking) though, and once matched, update the corresponding inspection_number field.

# URLs of the feature services
fs1_url = "https://services-ap1.arcgis.com/xyzxyzxyzxyzx/arcgis/rest/services/featureservice1/FeatureServer/0"
fs2_url = "https://services-ap1.arcgis.com/xyzxyzxyzxyzx/arcgis/rest/services/featureservice2/FeatureServer/0"

import datetime
from arcgis.gis import GIS
from arcgis.features import FeatureLayer
import certifi
import urllib3
import ssl
import warnings
from urllib3.exceptions import InsecureRequestWarning

# Create a default SSL context with certificate verification
ssl_context = ssl.create_default_context(cafile=certifi.where())
http = urllib3.PoolManager(ssl_context=ssl_context)

# Make a request to verify the setup
response = http.request('GET', 'https://myorg.arcgis.com')
print("http response: " + str(response.status))

# Suppress only the single InsecureRequestWarning from urllib3 if necessary
warnings.simplefilter('ignore', InsecureRequestWarning)

# Create GIS object
print("Connecting to AGOL")
client_id = 'xyzxyzxyzxyzx'
client_secret = 'xyzxyzxyzxyzx'

gis = GIS("https://myorg.arcgis.com", client_id=client_id, client_secret=client_secret)
print("Logged in as: " + gis.properties.user.username)

# Access the feature layers
fs1_layer = FeatureLayer(fs1_url)
print(fs1_layer)
fs2_layer = FeatureLayer(fs2_url)
print(fs2_layer)

#def update_feature_services(fs1_layer, fs2_layer):
# Mapping of inspection_number values to field names in fs2
inspection_mapping = {
  '2 weeks': 'insp_2_weeks',
  '4 weeks': 'insp_4_weeks',
  '6 weeks': 'insp_6_weeks',
  '8 weeks': 'insp_8_weeks',
  '10 weeks': 'insp_10_weeks',
  '12 weeks': 'insp_12_weeks',
  '4 months': 'insp_4_months',
  '5 months': 'insp_5_months',
  '6 months': 'insp_6_months',
  '9 months': 'insp_9_months',
  '12 months': 'insp_12_months',
  '15 months': 'insp_15_months',
  '18 months': 'insp_18_months',
  '21 months': 'insp_21_months',
  '24 months': 'insp_24_months'
}

# Query to get all features from fs1 without geometry
fs1_features = fs1_layer.query(where="inspection_number <> 'Ad-hoc Inspection'", out_fields=['objectid', 'globalid', 'date_of_inspection', 'plantation', 'years_planted', 'username', 'inspection_number'], return_geometry=False).features
print(f"Input features found")

# Query to get all features from fs2 without geometry
fs2_features = fs2_layer.query(out_fields="*", return_geometry=False).features
print(f"Target features found")

# Iterate over each feature in fs1
for fs1_feature in fs1_features:
  attributes = fs1_feature.attributes
  plantation = attributes['plantation']
  years_planted = attributes['years_planted']
  inspection_number = attributes['inspection_number']
  username = attributes['username']
  print(f"
Plantation: {plantation}, PYear: {years_planted}, Inspection: {inspection_number}, User: {username}")

# Convert date_of_inspection to string and extract the date part
  if attributes['date_of_inspection']:
      inspection_date = datetime.datetime.fromtimestamp(attributes['date_of_inspection'] / 1000).strftime("%Y/%m/%d")
      print(f"Checking {username} on {inspection_date}")

# Find the corresponding feature in fs2
      for fs2_feature in fs2_features:
          fs2_attributes = fs2_feature.attributes
          if fs2_attributes['Plantation'] == plantation and fs2_attributes['PlantingYear'] == years_planted:
              # Get the corresponding field name in fs2
              field_name = inspection_mapping.get(inspection_number)
              print(f"Inspection: {field_name}")
              if field_name and not fs2_attributes[field_name]:
                  # Update the matching field with 'username on date' text
                  fs2_attributes[field_name] = f"{username} on {inspection_date}"
                  fs2_layer.edit_features(updates=[fs2_feature])
                  print(f"Updated {username} on {inspection_date}")

I feel like maybe this would be better done using arcpy.da.searchcursor or something like that, but I'm not proficient in wielding that code yet.

Visual representation of the data below (the dates don't align, but you should get the idea). Repeated inspections should trigger updates to a series of fields in a row.

LindsayRaabe_FPCWA · Answer

I feel like this was close, but it was more advanced than I'm used to wielding and couldn't make progress. I'm taking a different approach that I can follow easier, though it may not be as efficient as other methods! Will start a new post to simplify the question to it's core.

Hayden Welch · Answer

As a sidenote, I mostly use arcpy for my work, so I might have messed up some of the boilerplate for dealing with FeatureLayer objects. My goal here was to flatten your loop and make it so you're only doing one request per feature in fc2. Plus the data frame request for fc1.

Because there's no early return condition/continue condition in your original code, you are pinging the server len(fc1)*len(fc2) times. Which with a large dataset could be in the hundreds of thousands of requests.

I'm also not sure if you need to save your edits even applying changes to a hosted layer through the API.

A final note, if there is an interface for updating a feature layer with a data frame, you could get this whole operation completed with 3 requests (pull fc1 into data frame, pull fc2 into data frame, apply updates to fc2_df and push to server).

Hayden Welch · Answer

EDIT: After looking into it more, we can cache the updates locally and apply them all at once which will greatly speed up the script (len(fc2) less edit requests), Changes are applied below by adding an `updates` list

Your flow seems to be correct, You are doing a lot of querying though. I spent some time to leverage the `as_df` flag on fc1 so you aren't sending network requests for every single feature:

# URLs of the feature services
fs1_url = "https://services-ap1.arcgis.com/xyzxyzxyzxyzx/arcgis/rest/services/featureservice1/FeatureServer/0"
fs2_url = "https://services-ap1.arcgis.com/xyzxyzxyzxyzx/arcgis/rest/services/featureservice2/FeatureServer/0"

import datetime
from arcgis.gis import GIS
from arcgis.features import (
    FeatureLayer, 
    Feature, 
)
from pandas import DataFrame
import certifi
import urllib3
import ssl
import warnings
from urllib3.exceptions import InsecureRequestWarning

# Create a default SSL context with certificate verification
ssl_context = ssl.create_default_context(cafile=certifi.where())
http = urllib3.PoolManager(ssl_context=ssl_context)

# Make a request to verify the setup
response = http.request('GET', 'https://myorg.arcgis.com')
print("http response: " + str(response.status))

# Suppress only the single InsecureRequestWarning from urllib3 if necessary
warnings.simplefilter('ignore', InsecureRequestWarning)

# Create GIS object
print("Connecting to AGOL")
client_id = 'xyzxyzxyzxyzx'
client_secret = 'xyzxyzxyzxyzx'

gis = GIS("https://myorg.arcgis.com", client_id=client_id, client_secret=client_secret)
print("Logged in as: " + gis.properties.user.username)

# Access the feature layers
fs1_layer = FeatureLayer(fs1_url)
fs2_layer = FeatureLayer(fs2_url)

#def update_feature_services(fs1_layer, fs2_layer):
# Mapping of inspection_number values to field names in fs2
inspection_timeframes = [
    '2 weeks',   '4 weeks',   '6 weeks', 
    '8 weeks',   '10 weeks',  '12 weeks',

'4 months',  '5 months',  '6 months', 
    '9 months',  '12 months', '15 months', 
    '18 months', '21 months', '24 months',
]

# Because the field names are consistent, we can dynamically generate the mapping from the inspection_timeframes
inspection_mapping = {
    timeframe: f"insp_{timeframe.replace(' ', '_')}"
    for timeframe in inspection_timeframes
}

# Declare fields and where clause for fs1
out_fields = ['objectid', 'globalid', 'date_of_inspection', 'plantation', 'years_planted', 'username', 'inspection_number']
where_clause = "inspection_number <> 'Ad-hoc Inspection'"

# Pull fs1 as a DataFrame so we only need to iterate fs2
fs1_df: DataFrame = fs1_layer.query(where=where_clause, out_fields=out_fields, return_geometry=False, as_df=True)

# Query to get all features from fs2 without geometry
fs2_features: list[Feature] = fs2_layer.query(out_fields="*", return_geometry=False).features

# Flag for using the first feature if multiple matches are found
use_first = True

# Initialize the update list
updates = []
# Iterate through fs2 features and update the matching field
for fs2_feature in fs2_features:
    fs2_attributes = fs2_feature.attributes

# Get identifying attributes from fs2
    years_planted = fs2_attributes['PlantingYear']
    plantation = fs2_attributes['Plantation']

print(f"
Plantation: {plantation}, PYear: {years_planted}")

# Skip if plantation or years_planted is empty, can't get a match without both
    if not plantation or not years_planted:
        print(f"	[WARNING] {plantation=} | {years_planted=} for feature {fs2_attributes['OBJECTID']}")
        continue

# Query fs1_df for the matching feature
    fs1_query = fs1_df.query(f"plantation == '{plantation}' and years_planted == {years_planted}")

# Skip if no matching feature is found
    if fs1_query.empty:
        print(f"	[WARNING] No match found for plantation {plantation} and planting year {years_planted}")
        continue

# Warn if multiple matching features are found
    if len(fs1_query) > 1:
        print(f"	[WARNING] Multiple features found for plantation {plantation} and planting year {years_planted}")
        if not use_first:
            print("	[WARNING] Skipping feature")
            continue

# Get the matching feature
    fs1_feature = fs1_query.iloc[0]
    inspection_number = fs1_feature['inspection_number']
    username = fs1_feature['username']
    inspection_date = datetime.datetime.fromtimestamp(fs1_feature['date_of_inspection'] / 1000).strftime("%Y/%m/%d")

# Skip if the field name is not found (Use walrus operator to assign and check in one operation)
    # NOTE: .get() will raise an error if the key is not found, so we set a default value of None
    if not (field_name := inspection_mapping.get(inspection_number, None)):
        continue

# Skip if the field is already populated
    if fs2_attributes[field_name]:
        continue

print(f"
	Plantation: {fs2_attributes['Plantation']}, PYear: {fs2_attributes['PlantingYear']}, Inspection: {inspection_number}, User: {username}")
    print(f"	Checking {username} on {inspection_date}")

# Update the matching field with 'username on date' text
    print(f"	Inspection: {field_name}")
    fs2_attributes[field_name] = f"{username} on {inspection_date}"
    updates.append(fs2_feature)
    print(f"	Updated {username} on {inspection_date}")

# Apply the update list
if updates:
    print(f"	Applying updates for {len(updates)} features")
    fs2_layer.edit_features(updates=updates)
else:
    print("All features up to date")