Append feature set to SDE

[ARyanWADNR]

Hi All!

I have been struggling with this one for a few days now, and can't figure out what is going wrong. I am automating an append function that I now do manually (using a selection from a feature service in ArcPro and then appending to SDE) with python instead.

The goal is to use the OIDs for the recent features to query the feature service, and then append the resulting feature set to an SDE feature class. The query works perfectly, and I get a feature set, I have then converted the ArcGIS API for Python version of the feature set to the arcpy version so that I can append the records. 

Once I get to the append I receive the error code 160250: Objects in this class cannot be updated outside an edit session Failed to execute (Append). Even when working in an editing session. 

Here is the bare bones of what I am doing: 

# Get the Feature set using Python API query

# Oids are gathered as a list
imputStringOids = []

#Connect to ArcGIS online and get the feature service
gis = GIS('home')
fs = gis.content.get('XXXXXXXXXXXXXXXXXXx')
add_Vssl = fs.layers[0]

# Query the feature service using the oids once they have been converted to a string
oids = "OBJECTID in (" + str(imputStringOids)[1:-1] + ")"
vsslDetailsQuery = add_Vssl.query(where=oids)
# Checks to see what format the result is in, its always <FeatureSet> X number of features
vsslDetailsQuery 

# Convert the API feature Set to a arcpy Feature set ( this is one step I am not very confident in).
feature_set_arcpy = arcpy.FeatureSet(vsslDetailsQuery)

#Begin editing session and append
workspace = os.path.dirname("X:\xyz\xyzx\xyzxy")
edit = arcpy.da.Editor(workspace)
edit.startEditing(with_undo=True, multiuser_mode=True)
edit.startOperation()

arcpy.management.Append(inputs=feature_set_arcpy, 
        target=dvrp_Feature_Class, schema_type="NO_TEST", field_mapping= CRAZY LONG FIELD MAPPING, subtype="", expression="", match_fields=None, update_geometry="NOT_UPDATE_GEOMETRY")

edit.stopOperation()
edit.stopEditing(True)

Other possibly useful info:

• The Oids are being collected from emails that come in when a new record is recorded
• The append is going into a traditionally versioned SDE feature class within an Oracle database
• I am open to changing this whole method up if there is a better way to do it, my other idea was to match my current workflow by using a map object to do the selection of the oids and then the append, but the feature set seems like it should be an easier method.

Thank you for any help you can give me!!!

Andrew

[RPGIS]

Hi @ARyanWADNR,

So the append tool can sometimes be problematic when running outside of a pro session and can lead to issues. One recommendation that I have is instead to use cursors to update the feature rather than calling a tool. Cursors give you more control over certain conditions and are better at limiting certain types of information.

import arcpy

from arcpy.da import search as SearchCursor, update as UpdateCursor, insert as InsertCursor, Edit as Editor

from arcpy import ListFields

gis = GIS('home')
fs = gis.content.get('')
layer = fs.layers[0]

fields = [field.name for field in ListFields(layer ) ]
# fields = [field.name for field in ListFields(layer ) if <some matching field conditions> such a list of names or datatype, etc... ]

data = {}
with search( layer , fields ) as cursor:
    for row in cursor:
        data[row[0]] = row
with Edit(<your editing layer workspace>) as editing:
    with insert(<editing layer>, fields ) as cursor:
        for key, values in data.items():
            cursor.insertRow(values)

This is just an example but should at least perhaps help guide you in case you want to try a different approach. With cursors, as mentioned above, these give you more control on how to set certain updates depending on your criteria and are far more flexible than the imported tools.

@HaydenWelch, @DanPatterson , @DougBrowning  may provide better recommendations or even other methods that yield the same result.

[DougBrowning]

I have not had to use a edit session in a long time.

I would try this vs 'home' as it grabs credentials from Pro

gis = GIS('pro')

Also I usually just give it the full URL to the REST service with /layernumberyouwant as an in source and it works great.  

For Field Mapping append will append any fields that match for you.  I gave up on field mapping.  If the fields names are different i copy the FC into memory then add field, then calc over then append.

I much prefer append to cursor.  It takes care of any projection issues and if I add any fields to both sides it just keeps on working.  Cursor requires a code change for any schema change.  Harder to maintain.

Hope that helps

[HaydenWelch]

Cursors don't require code change on schema change. If you just make sure that you only insert matching/mapped (as in my example) fields, you can get away with it easily. I will sometimes just use a basic field expansion script if I want to bring over non-matching fields:

from arcpy import Field, ListFields
from arcpy.da import SearchCursor
from arcpy.management import AddField
def add_field(target: str, field: Field) -> None:
    AddField(
        target,
        field_name=field.name,
        field_type=field.type,
        field_precision=field.precision,
        field_scale=field.scale,
        field_alias=field.aliasName,
        field_domain=field.domain,
        field_is_nullable=field.isNullable,
        field_is_required=field.required,
        field_length=field.length,
    )
    
def sync_fields(source: str, target: str) -> None:
    source_fields: set[str] = set()
    target_fields: set[str] = set()
    with SearchCursor(source, '*') as c:
        source_fields = set(c.fields)
    with SearchCursor(target, '*') as c:
        target_fields = set(c.fields)
    _missing = source_fields - target_fields
    if not (_missing := source_fields - target_fields):
        return
    for field in [f for f in ListFields(source) if f.name in _missing]:
        add_field(target, field)

Not to say the way you're doing it is wrong, but you can definitely use cursors in a way that makes schema changes more bearable. I feel it's worth it for the clarity and being able to handle operations on a per-row basis.

[ARyanWADNR]

Do you know why an append from a Feature Set to an SDE feature class isn't working? I have been hoping someone would double check that my basic idea of using a Feature Set from a query to an internal SDE was workable or not.

Thanks!

[DougBrowning]

It works we do it.  See my post on using a full URL instead.  Then what I do is MakeFeatureLayer then SelectionByAttribute then append.

[HaydenWelch]

A couple things

1.  You need to make sure that you pull the full shape using a cursor otherwise you will end up with invalid geometry
2. You can grab fields directly from a cursor without having to use the really slow ListFields function
3. You can stack cursors in a single with block so you don't have the overhead of storing the entire table in memory (can be an issue with larger datasets)

Note on 3: This method will only work with cursors on different objects, don't try to initialize a search and insert cursor on the same object in the same context, for that you do need an intermediate container.

from arcgis import GIS
from arcpy.da import (
    SearchCursor,
    InsertCursor,
    Editor,
)

gis = GIS('pro')
fs = gis.content.get('')
source = fs.layers[0]
target = fs.layers[-1]
target_workspace = 'target_workspace'

# Grab fields from a cursor
with SearchCursor(source, '*') as c:
    source_fields = set(c.fields)
with SearchCursor(target, '*') as c:
    target_fields = set(c.fields)

# Use set math to find intersection of fields
# Make sure to pull full shape!!
fields = ['SHAPE@'] + list(source_fields & target_fields)

# Get cursor context and insert the rows
with (
    Editor(target_workspace),
    SearchCursor(source, fields) as s_cur,
    InsertCursor(target, fields) as i_cur
):
    for row in s_cur:
        i_cur.insertRow(row)

If you want to do field mapping, you can use a simple dictionary map and a dict.get:

...
f_map = {
    'f1': 'f2',
    'f4': 'f6',
}

# Get cursor context and insert the rows
with (
    Editor(target_workspace),
    SearchCursor(source, fields) as s_cur,
    InsertCursor(target, [f_map.get(f, f) for f in fields]) as i_cur
):
    for row in s_cur:
        i_cur.insertRow(row)

[RPGIS]

That was going to be my other recommendation as well. Having a dictionary of fields to map makes data migration easier but it is a bit time consuming if you only know one dataset and how to map to it. If I have multiple datasets I typically use a dictionary such as the one below.

fieldmappings = {
    ('A_FieldA','B_FieldA','C_FieldA') : 'MyFieldA',
    ('A_FieldB','B_FieldB') : 'MyFieldB'
    }

[ARyanWADNR]

I'll give it a try. I dont have much experience with cursors, so I never really think about them.

Thank you for the response, I will update once I've tried this method out.

[RhettZufelt]

I have done this quite a bit, but I don't bother with the featureset (other than to query it)

I get a [List] of the OID's (I use GlobalID's, but OID's should work fine) of the features from the hosted feature service, then pass that 'expanded' list as the Where clause in arcpy.management.Append.

Post = r'\\computer\path\database.sde\owner.SCHEMA.Posts'


item_id = '3b61fd5e5e9842dc80sdfsd443465'  # Hosted Post data
item = portal.content.get(item_id)

lyr1 = item.layers[1]   # Portal Features

AddPostList = ['123','124','125',....]  #  List of GlobalID's to append to SDE data

with arcpy.EnvManager(preserveGlobalIds=True, maintainAttachments = True,  scratchWorkspace=r"\\path\to\Default.gdb", workspace=r"\\path\to\Default\Default.gdb"):
    Post = arcpy.management.Append(inputs=[lyr1.url], target=Post, schema_type="NO_TEST", subtype="", expression="GlobalID IN ('" + '\',\''.join(AddPostList) + "')")[0]

Below, you can see how the list gets expanded:

I have stripped the field mapping from this code as it needs the full path to the HFS fields, so it was too much and confusing to post.

R_