Calculate Field using a layer on Portal - Fails every time

1105
7
Jump to solution
10-20-2021 08:32 AM
KBKnight
New Contributor III

Hi all,

I am trying to do a (relatively) simple thing. I have a polygon dataset on AGOL. I need to update a field with new values. It's not possible to overwrite from Pro because it's used in Field Maps and synced (replicas issues). So, I have tried the following:

1. load layer from Portal in Arc Pro (works great!)

2. Join to table with updated field (no problem)

3. Calculate the field in the polygon dataset to be the field from the joined table (fails every time after a LOOOONG run time (maybe hours) of the progress bar slowly moving.

I've tried using a smaller, sample dataset and this still doesn't work. I CAN manually change an entry for the field in question so changing one row's data works fine. The field is text with a domain applied, but otherwise nothing special at all. 

Next steps? Things I could try? Workarounds?

0 Kudos
1 Solution

Accepted Solutions
jcarlson
MVP Esteemed Contributor

 

from arcgis.gis import GIS
from arcgis.features import FeatureCollection
import pandas as pd

# Log in. Will prompt for password using getpass
gis = GIS('your-portal-url', 'user')

# Get polygon layer
lyr = gis.content.get('polygon-layer-itemID').layers[0] # or whatever the layer index is

# Create dataframe from layer; using a query limits output fields and geometry for performance
l_df = lyr.query(as_df=True, out_fields=['the-field', 'identifier'])

# Get the table
tbl = pd.read_csv('path-to-csv')

# Merge dataframes; if field names are identical, use on='identifier'
df = l_df.merge(tbl, left_on='identifier', right_on='matching-identifier')

# Replace field values
df['the-field'] = df['the-other-field']

# Submit updates in batches
n = 0
while n < len(df):
    fs = df.iloc[n:n+200].spatial.to_featureset()
    lyr.edit_features(updates=fs)
    n += 200

 

 

Notes:

  • If your table is a feature class somewhere, you can try using arcgis.features.GeoAccessor.from_featureclass('path-to-fc')
  • The edit_features function will automatically ignore any fields not present in the published layer, so if your dataframe has ancillary fields from the merge, don't worry about it. As long as the field being updated and the ObjectID / GlobalID match, the edit will apply
  • The API docs suggest using the append function for larger edits. I admittedly wrote a version of this script prior to 10.8.1, when that function was unavailable to Enterprise users, hence the 200-row batch edits. I've not gone back to test it on the new function. But it's probably worth it to look into.

Good luck! I haven't edited a layer quite as large as yours, but updating the attributes on a layer w/ about 60k rows only took a few minutes. Since you're not adjusting the geometry of the layer, it should be pretty fast.

- Josh Carlson
Kendall County GIS

View solution in original post

0 Kudos
7 Replies
KBKnight
New Contributor III

Forgot to mention, I am using Arcade as the Expression Type for this.model_example.JPG

0 Kudos
jcarlson
MVP Esteemed Contributor

How many records are in the table? And is the joined table just a local file, feature class, etc?

Are you comfortable working with Python? It would be quite easy (and probably faster) doing this via a spatially enabled dataframe. I've had similar situations with field calculations on published layers running quite a while, but a bit of Python can handle it really well.

- Josh Carlson
Kendall County GIS
0 Kudos
KBKnight
New Contributor III

The FULL polygon dataset and table are large (130k entries), but I have tried on a subset of about 5,000 and it still fails. 

I'm medium on Python. I can update, change, and tinker with Python scripts and cut and paste stuff together pretty well. If you have an example script, I can probably (with considerable cursing) modify it to work with my data. 

0 Kudos
jcarlson
MVP Esteemed Contributor

 

from arcgis.gis import GIS
from arcgis.features import FeatureCollection
import pandas as pd

# Log in. Will prompt for password using getpass
gis = GIS('your-portal-url', 'user')

# Get polygon layer
lyr = gis.content.get('polygon-layer-itemID').layers[0] # or whatever the layer index is

# Create dataframe from layer; using a query limits output fields and geometry for performance
l_df = lyr.query(as_df=True, out_fields=['the-field', 'identifier'])

# Get the table
tbl = pd.read_csv('path-to-csv')

# Merge dataframes; if field names are identical, use on='identifier'
df = l_df.merge(tbl, left_on='identifier', right_on='matching-identifier')

# Replace field values
df['the-field'] = df['the-other-field']

# Submit updates in batches
n = 0
while n < len(df):
    fs = df.iloc[n:n+200].spatial.to_featureset()
    lyr.edit_features(updates=fs)
    n += 200

 

 

Notes:

  • If your table is a feature class somewhere, you can try using arcgis.features.GeoAccessor.from_featureclass('path-to-fc')
  • The edit_features function will automatically ignore any fields not present in the published layer, so if your dataframe has ancillary fields from the merge, don't worry about it. As long as the field being updated and the ObjectID / GlobalID match, the edit will apply
  • The API docs suggest using the append function for larger edits. I admittedly wrote a version of this script prior to 10.8.1, when that function was unavailable to Enterprise users, hence the 200-row batch edits. I've not gone back to test it on the new function. But it's probably worth it to look into.

Good luck! I haven't edited a layer quite as large as yours, but updating the attributes on a layer w/ about 60k rows only took a few minutes. Since you're not adjusting the geometry of the layer, it should be pretty fast.

- Josh Carlson
Kendall County GIS
0 Kudos
KBKnight
New Contributor III

This worked! Thank you. I will mark it as a solution! For any future readers, I found one small typo: on line 12, there needs to be an "=" after out_fields. But, your solution was brillian @jcarlson! Thank you so much!

0 Kudos
jcarlson
MVP Esteemed Contributor

Good catch! I'll edit my post.

- Josh Carlson
Kendall County GIS
0 Kudos
DominicRoberge2
Occasional Contributor III

Hey Kevin

 

could you try to use a Notebook in ArcGIS Pro? I do something similar but my source is a AGOL hosted feature layer. 

DominicRoberge2_0-1634744952438.png

 

0 Kudos