Select to view content in your preferred language

Layers Disappear after running Arcpy Notebook with Edit Session, and Nested Update Cursors

727
2
10-14-2022 10:03 AM
Labels (3)
RWATT
by
Occasional Contributor

Hello,

I'm working on an ArcGIS Pro (v3.0.2) notebook that copies attributes between a point (1) and a line (2) and then copies the line feature (2) to an identical line feature (3). 

Details on each of these features:

1. A Feature Service Feature Class of Survey Entries from Survey123. They have important attributes from the survey, but no meaningful geographic component.

2. An Enterprise Geodatabase in PostgreSQL (v10.6.1) Feature Class of lines. This feature class is receiving the attributes from 1.

3. A Feature Service Feature Class of lines. This feature class recieves the entire feature from 2 via Copy>Paste Special, or arcpy.management.Append.

This process works consistently and without error messages.

However, these code cells will occasionally cause some or all layers in my map and corresponding layout to "disappear". 

These layers remain in the Contents pane, I can still access their attribute tables, and sometimes I can still select them by clicking in the map.

Their visibility is restored by any of the following:

1. Closing and reopening the map and layout.

2. Removing the layer from the Contents and re-adding it.

3. Clearing the cache via Project>Options>Display>"Clear Cache Now"

4. Closing and reopening the project. 

Their visibility is NOT restored by: 

1. Refreshing the map in the lower-right corner.

2. Clearing the layer cache in the contents pane.

3. Refreshing the database connection

4. Clearing the workspace environment. arcpy.ClearEnvironment('workspace')

5. Deleting the workspace. del arcpy.env.workspace, workspace

6. Clearing the workspace cache. arcpy.management.ClearWorkspaceCache()

 

I think this is likely an issue with the database connection, but I'm out of ideas on how to avoid it. Any suggestions are appreciated. Thanks!

Three Code Cells:

 

#This ipynb is built for use in ArcGIS Pro to produce new digital service cards from Survey123 and service points collected in the field. 

#This cell sets up the context for future cells

#imports. getpass is to pull the windows username for the PDF details at the end.
import arcpy, getpass

#define the project, map, mapview, layout, map frame in the layout, survey entry layer, and workspace.
aprx = arcpy.mp.ArcGISProject('CURRENT')
m = aprx.listMaps("Editing_Map")[0]
mv = aprx.activeView
layout = aprx.listLayouts("CombinedLayout")[0]
map_frame = layout.listElements("MAPFRAME_ELEMENT", "Map Frame")[0]
lyr = m.listLayers("Survey_Entries")[0]
#arcpy.env.workspace = r"C:\Users\USER\Documents\ArcGIS\Projects\Service Cards\arcgis(2).sde"

 

 

 

#This cell will:
    #validate that only a single gisdb service is selected, and:
        #Pull all of the survey content into the gisdb service feature based on matching Work Order #
        #Pull the Service ID from the gisdb service to the Survey point (We use this to track which surveys have been processed)
        #Copy the entire gisdb service feature into the AGOL layer.
    
workspace = r"C:/Users/USER/Documents/ArcGIS/Projects/Service Cards/arcgis(2).sde"

#Check that only one record is selected in the gisdb services layer
if int(arcpy.management.GetCount("gisdb.sde.Service").getOutput (0)) == 1:
    
    #start an edit session
    edit = arcpy.da.Editor(workspace)
    
    #Start editing. I don't need to undo in this process and the workspace is versioned.
    edit.startEditing(with_undo=False, multiuser_mode=True)
    
    edit.startOperation()
    
    #Sync services to survey.

    ## field names in the service feature class. Many have been removed for brevity
    service_fields = ['w_o_','city','installati', 'premises', 'house_no', 'street',
                       'lot_no', ]

    ## field names in the survey feature class. Many have been removed for brevity.
    survey_fields = ['USER_WO_Number', 'USER_Town', 'USER_Date_Completed', 'premise_ids', 'house_number',
                     'street_name', 'apt_unit']

    ##select the service and survey feature classes
    service_lines = m.listLayers("gisdb.sde.Service")[0]
    survey = lyr

    #Create update cursor "service_cursor" for services
    with arcpy.da.UpdateCursor(service_lines, service_fields) as service_cursor:
        #iterate through lines/features/services in service_lines
        for service_row in service_cursor:
            #Create update curor "service_cursor" for survey points
            with arcpy.da.UpdateCursor(survey, survey_fields) as survey_cursor:
                for survey_row in survey_cursor:
                    ## find the entry with the same WO ID
                    if service_row[0] == survey_row[0]: ## WO
                        ## set each field in the service layer equal to the value in the 
                        ## corresponding field from the survey layer

                        ## Towns formatted to fit in server domain
                        if "" in survey_row[1]:
                            service_row[1] = ""
                        elif "" in survey_row[1]:
                            service_row[1] = ""
                        elif "" in survey_row[1]:
                            service_row[1] = ""
                        elif "" in survey_row[1]:
                            service_row[1] = ""
                        elif "" in survey_row[1]:
                            service_row[1] = ""
                        elif "" in survey_row[1]:
                            service_row[1] = ""
                        elif "" in survey_row[1]:
                            service_row[1] = ""
                        elif "" in survey_row[1]:
                            service_row[1] = ""
                        else:
                            service_row[1] = survey_row[1].replace("_", " ")

                        service_row[2] = survey_row[2].replace(hour=12) ## completion date
                        service_row[3] = survey_row[3] ## premises
                        service_row[4] = survey_row[4] ## house number
                        service_row[5] = survey_row[5].title() ## street
                        service_row[6] = survey_row[6] ## lot/unit
                        service_row[7] = survey_row[7] ## size
                        service_row[8] = survey_row[7] ## size
                        service_row[9] = survey_row[8] ## material - text

                        if survey_row[8] == 'Steel': ## if the survey has material as steel
                            service_row[10] = 3 ## then set service material to 3, which is coded as steel
                        else:
                            service_row[10] = 5 ## otherwise set it to 5 which is plastic

                       #Many attributes have been removed for brevity and privacy here.

                        survey_row[37] = service_row[46] ## service ID

                        #save the service ID to a new variable for use in the next cell.
                        service_input = service_row[46]
                        
                        service_cursor.updateRow(service_row)
                        survey_cursor.updateRow(survey_row)
    
    # Stop the edit operation.
    edit.stopOperation()

    # Stop the edit session and save the changes
    edit.stopEditing(save_changes=True)
    
    #Tidying up that I thought might help with this error. It has had no apparant impact.
    arcpy.management.ClearWorkspaceCache()
    
    del edit
    
    del survey_fields, service_fields, service_lines, survey, service_cursor, survey_cursor
    
################################################################

    
   
    print("Survey2Services Complete!")
    

else:
    #print an error to remind the user to select only one service
    print('Select only 1 Service in the svc db layer and rerun this cell!')
#Check that only one service is selected
if int(arcpy.management.GetCount("gisdb.sde.Service").getOutput (0)) == 1:
#Copy selected svc database layer to AGOL layer.
    arcpy.management.Append("gisdb.sde.Service", "Services_AGOL", "TEST")

 

 

2 Replies
RWATT
by
Occasional Contributor

I think I might have found this bug in the issues addressed for v10.7 at https://downloads.esri.com/support/downloads/other_/107-IssuesAddressedList-03182019.pdf.

"BUG-000117721
High rate of edit operations and reconciles generates lock contention on the sde.states table in the version_util procedure new_edit_state".

I'm hoping an update to 10.7 or later will resolve this issue.

 

0 Kudos
RWATT
by
Occasional Contributor

I wrote a temporary workaround that minimizes edit operations on the database by doing all the editing locally and then just making a single edit to append the completed feature on the gdb. 

I will wait until I can run the original code on an upgraded version to accept this as the solution, in case there's something else at play here.