Unexpected update cursor behavior affecting feature linked annotation

[RichardFairhurst]

I have been using code to transfer data from one source to another using a da update cursor.  However, I got an unexpected and very upsetting behavior when I included a field in the update cursor field list that acts as a trigger to change the feature linked annotation when it is updated.  This occurred on a versioned SDE feature class that requires the use of an edit operation to let the update cursor work.

The field was only included in the update cursor's field list to provide a look up key to another table.  No value changes or duplications were written to that Key field by the update cursor.  The update cursor only read values from the trigger field.  Updated values were only written to the other fields in the update cursor field list. 

Unfortunately, when I applied the updateRow action to the cursor for each row, the feature linked annotation had its TextValue update process triggered even though nothing had been written directly to the trigger field.  I also noticed that when the update process triggered an annotation update that the update cursor sent the record through my record processing loop twice.  Double processing of a record could have a severe impact on many cursor update operations. However, in this particular case it was only an issue because it slows down the update process.

This behavior moved the annotation objects and overwrote annotation values that had been intentionally manually altered.  This has corrupted my annotation feature class and I now am scrambling to get a backup restored.  It never occurred to me that the field would act as a trigger for annotation just because it was included in the update cursor field list and read.  Why is that behavior so different from the manual editing process, which only triggers an annotation update when the trigger field is altered or duplicated?

I have modified the code to exclude all fields involved in the update of feature linked annotation, since I had another field that contained look up values related to the external table.  By excluding all fields that can trigger annotation updates the update cursor does affect the annotation of the feature class.  Also the records are processed just once by the cursor loop as well.  I had also used similar code on the same feature class before and only when the trigger field was in the update field list did the cursor trigger the annotation to update.

However, what if I had no alternative field availalbe and had to use the trigger field for the look up value?  Disabling the feature linked update behavior seems to be the only solution in that case,  I just wish I had known about this behavior before reconciling all of my versions.  Is this behavior adequately documented?  I don't recall reading or hearing about it.

Here is the code in case anyone thinks I must have actually written updated values to the trigger field.  The field that can trigger the feature linked annotation to update itself is called STNAME.

import arcpy
import os
from time import strftime


sourceFC = r"C:\Users\RFAIRHUR\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\Trans Connection to SQL Server.sde\MasterStreetName"
updateFC = r"C:\Users\RFAIRHUR\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\Trans Connection to SQL Server.sde\GDB_TRANS.TRANS.TRANSPORTATION_MAINT\CENTERLINE"


print "Make Feature Layer: " + strftime("%Y-%m-%d %H:%M:%S")
arcpy.MakeFeatureLayer_management(updateFC, "CENTERLINE_Layer")
arcpy.SelectLayerByAttribute_management("CENTERLINE_Layer", "NEW_SELECTION", "FULL_NAME_MIXED_CASE IS NULL")
records = int(arcpy.GetCount_management("CENTERLINE_Layer").getOutput(0))
while records >= 1:
    print str(arcpy.GetCount_management("CENTERLINE_Layer")) + " Records Selected: " + strftime("%Y-%m-%d %H:%M:%S")
    updateFC = "CENTERLINE_Layer"


    workspace = r"C:\Users\RFAIRHUR\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\Trans Connection to SQL Server.sde"


    sourceFieldsList = ["FULLNAME_ALL_CAPS", "FULLNAME"]  

    updateFieldsList = ["STNAME", "FULL_NAME_MIXED_CASE"]

    valueDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, sourceFieldsList)}  


    edit = arcpy.da.Editor(workspace)  
    edit.startEditing(False, True)   
    edit.startOperation()


    i = 0  
    with arcpy.da.UpdateCursor(updateFC, updateFieldsList) as updateRows:  
        for updateRow in updateRows:  
            i += 1
            if i % 101 == 0:
                break
            keyValue = updateRow[0]  
            if keyValue in valueDict:  
                for n in range (1,len(sourceFieldsList)):    
                    updateRow = valueDict[keyValue][n-1]  
                updateRows.updateRow(updateRow)  

    del valueDict   
    edit.stopOperation()  
    edit.stopEditing(True)
    arcpy.SelectLayerByAttribute_management("CENTERLINE_Layer", "NEW_SELECTION", "FULL_NAME_MIXED_CASE IS NULL")
    records = int(arcpy.GetCount_management("CENTERLINE_Layer").getOutput(0))
print "Finished All Records: " + strftime("%Y-%m-%d %H:%M:%S")

[RichardFairhurst]

I have a help ticket open with Esri to try to get them to log this as a bug.  I have demonstrated that just by including a field in the update cursor field list that is part of the label expression trigger for the feature linked annotation and updating a row inside an edit session that the annotation updates itself.  This occurs even if all I do is open an edit session, open an update cursor and use updateRow on a row without using the update cursor to modify or read any of the field values in the row.

So all update cursor updateRow events are treated by the Editor as though every Modify Record edit event included modifications to every field in the field list, even if no modifications of any kind were done.  When the field list includes a trigger field for the feature linked annotation the field is always considered to have been modified by the Editor when the updateRow event is processed and the row's annotation is always rebuilt to replace any prior manual annotation edits.

This is a severe bug or at least a behavior that requires a highly visible warning in the update cursor help so that users of feature linked annotation do not unknowingly use update cursors to trash months or years of manual annotation edits like I did.  If I did not have a backup of the annotation this incident would have been an utter disaster.

[RichardFairhurst]

It appears that the programmers of the update cursor believe that the behavior I have observed is intended, since they do not regard an update cursor in an edit session as needing to abide by the same rules as a normal interactive editor session.  So I have asked that this be treated as a documentation bug, because I do not believe that the documentation contains enough information for a developer to ever anticipate this behavior might occur so that they can avoid it.  This behavior is a severe gotcha based on my experience.

So the conditions required to trigger this behavior and cause your feature-linked annotation to regenerate itself according to its label engine rules are:

1.  An Editor session has been started by the Python code.

2.  The Update cursor includes a field list that has one or more trigger fields in it related to the feature-linked annotation classes label expression or SQL definition.

3.  The annotation already exists and has been manually edited to override some aspect of the default label appearance for the feature being processed by the cursor.

4.  The updateRow method is used on the row object of that feature, whether or not anything else was done to that row.

Feature Linked Annotation requires an Editor session to use the update cursor, so you cannot avoid it by not using an editor session.

Ways to avoid it include:

Prior to using an update cursor, run the data through a Search Cursor to read all of the fields that can trigger an annotation update into a dictionary that uses the ObjectID as the dictionary key (assuming that the ObjectID is not a trigger field).  While processing the update cursor access the trigger field values for the record currently being read from the dictionary to evaluate each row and only use updateRow on an update cursor field list that has fields that are not annotation trigger fields.

Use the Field Calculator to duplicate the field values of the trigger field(s) into a new field(s) that does not participate in the annotation label definition and use that new field(s) in the update cursor field list.

Duplicate just the features into a new feature class in a file geodatabase with any normal copying tool and preserve a unique ID for each record that does not participate in an annotation label expression that ties to the original source data.  Make all modifications using the fields that participate in the label expressions in this copy.  Use a dictionary and cursor to transfer the data through the unique ID field value and do not include any field that participates in a label expression in the update cursor field list when transferring the other updates to the source data.

Bottom line is never include a feature linked annotation trigger field in connection with an update cursor, unless you intend to have the annotation label engine regenerate every annotation label associated with every feature processed by the updateRow method of the update cursor,  I suspect that Composite Networks would have the potential for the updateRow method of an update cursor to trigger its modified record update behaviors if fields involved in composite relationships are included in the update cursor field list, although I have not tested that and I do not know how much potential damage could occur as a result..

[FreddieGibson]

Hi Richard,

Quick question for ya. Why are you calling the updateRow method when nothing has been updated? Does this behavior you're reporting occur when you only call updateRow after you've verified that the row needs to be updated?

[RichardFairhurst]

The real code that destroyed my annotation actually read the value in the field that participates in the label expression and then based on that value actually updated a field that did not participate in the label expression.  When the UpdateRow was called both fields were treated as modified, and the annotation was updated by the label engine.  That was unexpected, since I had made no changes to the value in the field that affected the annotation label expression, I had just read it.

My further tests where I did not read or update any values in any field have just been designed to show that the behavior occurs as soon as any row is passed to the updateRow method that in any way includes a feature linked annotation label expression/SQL field in the update cursor field list, no matter what else you do or don't do with the cursor.  So my tests have shown that including code to read the field value or excluding code to read the field value made no real difference to the behavior, it was the fact that it was in the field list at all when the updateRow method was processed.

In any case, it is only possible to do updates with an update cursor on feature linked annotation that fully avoids this behavior if you never ever include any field that participates in an annotation label expression/SQL in the update cursor field list so that is completely inaccessible to the updateRow method of the update cursor entirely.

[FreddieGibson]

So I want to make sure I fully understand your situation. Let's say you have two fields, which we'll just say are A and B. A is just a standard field in the table, but B is tied to a relationship class with various triggers set on it. Are you saying that when you make changes to A you see that your relationship class is triggered in response to listening for changes to B?

[RichardFairhurst]

No, that is not quite right.  I am saying that if B is in the field list of the update cursor then it will trigger all modified field value listeners it has set up as soon as the updateRow method is run for any given row, even if you make no change of any kind to the field B data for that row.  Including B in the field list always triggers the modified value listeners of field B and those listeners can only be stopped by excluding it entirely from the update cursor field list scope (since you should always intend to run updateRow as some point with an update cursor).

Including field A in the field list or not including A in the field list has no effect if it is just a standard field that has no relationship class trigger/listeners.

So go ahead and use an update cursor as long as you only access field A with the update cursor.  But the moment you let the update cursor have access to field B within an Editor session, you are firing the field B listeners every time you update a row, whether you thought you were making changes to field B or not with your code.

I am mentioning this, because this behavior is not at all like an interactive Editor session.  Field B can be in a table view and you can click on the Field B value and copy it to your clip board, and you can change the value in any other field and save the edits without triggering the Field B listeners.  In an interactive Editor session only an actual change to the Field B value triggers the listener.  The update cursor behavior is the equivalent of the Editor always triggering the listener for the row your cursor was on just by opening Field B for viewing in a table view and pressing the save button.  Nothing else is required to trigger the listener when the update cursor behavior is occurring.

[IanBroad]

I just ran into a similar problem with duplicate rows.

Has this bug been resolved yet? I'm on 10.2.2.

What a major pain.

[PerryKesselbaum]

I've got a similar issue. I'm running 10.2.2 and for each row the updateCursor processes with a field annotation is derived from in the field list, instead of regenerating the annotation attributes after each updateRow call, mine is setting the SHAPE_Length and SHAPE_Area of the annotation feature to 0 (effectively vanishing the annotation for that feature). This has happened using unversioned data in a MS SQL SDE and a copy of the same data in a local file geodatabase. Triggering an annotation update through editing the feature in ArcMap will restore the correct annotation geometry and the annotation is visible again.

This only happens when running the script from a standalone Python file, setting up an edit session and using startEditing(False, False), startOperation(), stopOperation(), and stopEditing(True). When run through ArcMap's Python console, with en edit session started through the ArcMap interface, the cursor runs as expected and the annotation is fine.

Is this already logged as a bug? Was an official solution ever offered or discovered?