topic Re: Script crashes in Python Questions

Script crashes

BrianLomas — Sat, 11 Dec 2021 11:12:45 GMT

When I run the following script it just spins and spins then arcmap turns to "not responding". I'm trying to update a versioned parcel layer using a .mdb file. When I "quote" out the updatecursor everything seems to run fine and it seems to enter the edit session as well. Any ideas on what it could be? I'm stuck.

from time import strftime 
    
print "Start script: " + strftime("%Y-%m-%d %H:%M:%S") 
    
import arcpy
from arcpy import env  
     
sourceFC = r'M:\ParcelDatabase\...\Feb_2016_Ownership'
     
sourceFieldsList = ["PARCEL","SITUS", "CURRENT_OWNER", "CO_OWNER", "OWNER_MAILADDR", "OWNER_CITY_STATE", "OWNER_ZIPCODE", "TAX_DISTRICT", "TAX_CODE", "SEC", "TOWNSHIP", "RANGE", "LEGAL"]  
     
# Use list comprehension to build a dictionary from a da SearchCursor  
valueDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, sourceFieldsList)}  

arcpy.env.workspace = r"Database Connections\parcelpubdelete.sde" 

updateFC = r"Database Connections\...\ArcGISSDE.SDE.TaxParcel"

updateFieldsList = ['PARCELID','SITEADDRESS','OWNERNME1', 'OWNERNME2', 'PSTLADDRESS', 'PSTLCITY', 'ZIPCODE', 'TAX_DIST', 'TAX_CODE', 'SEC', 'TWN', 'RGE', 'PRPRTYDSCRP']  

edit = arcpy.da.Editor(r"Database Connections\parcelpubdelete.sde")
edit.startEditing(False, True)
edit.startOperation()    
print "entered edit session"

with arcpy.da.UpdateCursor(updateFC, updateFieldsList) as updateRows:  
     for updateRow in updateRows:  
         # store the Join value of the row being updated in a keyValue variable  
         keyValue = updateRow[0]  
         # verify that the keyValue is in the Dictionary  
         if keyValue in valueDict:  
             # transfer the values stored under the keyValue from the dictionary to the updated fields.  
             for n in range (1,len(sourceFieldsList)):  
                 updateRow = valueDict[keyValue][n-1]  
             updateRows.updateRow(updateRow)  
     
del valueDict

edit.stopOperation()
edit.stopEditing(True) 
     
print "Finished script: " + strftime("%Y-%m-%d %H:%M:%S")

Re: Script crashes

JoshuaBixby — Thu, 24 Mar 2016 20:11:42 GMT

Without an explicit error message, it is difficult to offer suggestions. I would add some additional print statements to see what part of the code in the update cursor is getting hung up.

Re: Script crashes

AdrianWelsh — Thu, 24 Mar 2016 20:15:49 GMT

Brian,

This may be extra picky but it looks like there are five spaces for the indent in line 27 (where it should be four). I would hope python code could get around that, but I don't know.

If not that then, do you need to start an edit session for UpdateCursor? I was not aware that you needed to (but I do not have a ton of experience using UpdateCursor

(oh, just answered my question about the Edit session and cursors looking at this thread: can't edit row with arcpy.da.updatecursor in arcmap)

Re: Script crashes

WesMiller — Thu, 24 Mar 2016 20:27:51 GMT

You may also want to consider the number of records your dealing with. I would try creating a copy (for backup) and let it run for a while.

Re: Script crashes

JamesCrandall — Thu, 24 Mar 2016 20:29:30 GMT

Double-check and go over the field lists you have and verify there's no discrepancies between them and the actual sources.

Also, are you using a Joined field to populate some other source? If so, make sure you fully qualify the field's joined name, not just it's alias or original source name. (just a guess here).

Re: Script crashes

RuthBowers — Thu, 24 Mar 2016 20:39:12 GMT

I'm not sure if this is the problem, but you probably shouldn't be using the name of the command "updateRow" for the name of a variable. For example, instead of

for updateRow in updateRows:

put

for r in updateRows:

Just wondering, because everything else looks okay at first glance.

Re: Script crashes

BrianLomas — Fri, 25 Mar 2016 13:13:32 GMT

I placed a few print statements and everything seemed to be running. So, I added a counter and the script is in fact running but it's only cycling through a record every second or so. We have over 10,000 rows that will have 12 different attributes changed, at this rate it will take entirely too long. When I run the script on a fdb or shp is runs in 2 seconds with the same number of records. I'm new to scripting so I'm not sure if there's something I'm doing wrong. Any suggestions?

Re: Script crashes

BrianLomas — Fri, 25 Mar 2016 13:17:39 GMT

I kept getting an error saying I couldn't edit the layer outside an edit session even if I was in an edit session in Arcmap. So, I added the edit operation to the code and it stopped tripping the error.

Re: Script crashes

BrianLomas — Fri, 25 Mar 2016 13:20:39 GMT

I double checked the specified fields and everything seems to be good to go... The plan is to join a mdb and a versioned fc then update the fc with the attributes of the mdb.

Re: Script crashes

JoshuaBixby — Fri, 25 Mar 2016 13:26:47 GMT

Is the time to update a record linear, i.e., does it stay constant over time or get progressively slower? There have been discussions on GeoNet about the degraded performance of update cursors over time on large record sets. That said, the performance usually starts out reasonable and gets worse over time. Also, 10,000 records isn't that large, but it is starting to enter the range where some people say they have issues. Also, are you using a Parcel Fabric? I have also heard of slow update performance with parcel fabric over regular feature classes in enterprise geodatabases.

Overall, I think you should open an incident with Esri Support. If it is taking 1 sec/record from the start, something else is going on either in the application or backend database.

Re: Script crashes

BrianLomas — Fri, 25 Mar 2016 14:03:45 GMT

Thanks for the info Joshua. I started it again this morning and it ran about 1,000/min til about 5,000 then slowed up a bit, at ~7000 it just stopped. There's no message and I can't click on anything, it just froze... We're not using parcel fabric.

Re: Script crashes

JoshuaBixby — Fri, 25 Mar 2016 15:38:18 GMT

OK. Seeing a parcel fabric is not involved, and the performance does degrade over time, this seems to generally fit into other discussions/complaints on GeoNet about update cursors on medium to large datasets.

A couple of ideas come to mind. First, can you go with multiuser_mode as False? If so, does that make a difference. Second, even though you have with_undo as False, you are still trying to update ~10,000 in one operation. You could use one edit session but chunk it up into multiple edit operations. In effect, iterate over the records committing the updates after every 1000-2000 records. Unfortunately with such an approach, the update cursor needs to be instantiated and released with each edit operation, so there is some work to ensure the records are ordered properly and divided up so records aren't missed.

Re: Script crashes

WesMiller — Fri, 25 Mar 2016 15:59:48 GMT

Not sure if this will work with version data, but you may be better off doing a join and select those records that have changed.

Workflow:

Add Join
for field in fields (on target)
select <> to see if attributes don't match
calculate field
next field repeat steps 3,4