cannot open file using arcpy.da.InsertCursor (randomly)

4043
14
Jump to solution
09-17-2020 08:59 AM
BrookeHodge
New Contributor III

Hi all, I'm hoping someone can help me a problem I can't figure out.  Below is my code.  I keep getting a RuntimeError: cannot open 'file I'm writing to' using a arcpy.da.InsertCursor.  It keeps locking for some reason, I do not have arc open when running (running it from PyCharm), and it will write to the file thousands of times... I am writing line segments to a feature class and it will do it thousands of times, but then seemingly randomly say it can't access it.  I have run it many times and it get's locked up in different places each time.  The only info I can find is that people need to close their cursors, but all my cursors (both search and insert cursors) are closed after each use.  I'm not sure what else to do.  Any ideas?  Description of what code is trying to do below if that is helpful at all as well as a screen shot of the error.

import arcpy

# Set environments / workspace.  Workspace should point to a gdb that has all the effort point feature classes
arcpy.env.workspace = r"C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb"
arcpy.env.overwriteOutput = True
# This is the input points
fc = r"C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb\EffortPoints_Prj_TEST_B"
# This is the output lines
sr = arcpy.Describe(fc).spatialReference
outputFC = arcpy.management.CreateFeatureclass(r"C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb", "EffortLines_TEST", "POLYLINE", "", "DISABLED", "DISABLED", sr)
# Add desired fields to outputFC
arcpy.AddField_management(outputFC, "MONTH", "TEXT")
arcpy.AddField_management(outputFC, "YEAR", "SHORT")
arcpy.AddField_management(outputFC, "FILEID", "TEXT")
arcpy.AddField_management(outputFC, "EVENTNO", "LONG")

# Make a feature Layer of the input points to work off.  This layer is temporary and will not persist after session ends unless saved
arcpy.MakeFeatureLayer_management(fc, "fc_lyr")

# Sort the table by fileid and eventno and create a new fc called fc_sort
sort_fields = [["FILEID", "ASCENDING"], ["EVENTNO", "ASCENDING"]]
fc_Sorted = arcpy.Sort_management("fc_lyr", "fc_sort", sort_fields)

# Create the search cursor
fields = ['FILEID', 'beauf_num', 'LEGTYPE', 'LEGSTAGE', 'vis_num', 'olviz_num', 'alt_num', 'LAND', 'MONTH', 'YEAR', "EVENTNO", "SHAPE@"]
with arcpy.da.SearchCursor(fc_Sorted, fields) as cur1:
    for row in cur1:
        # Name variables and assign values starting on first record of table
        fileid1 = row[0]
        beaufort = row[1]
        legtype = row[2]
        legstage = row[3]
        visiblty = row[4]
        oldvis = row[5]
        alt = row[6]
        land = row[7]
        month = row[8]
        year = row[9]
        eventno = row[10]
        # Get the geometry of the point for that record
        geom1 = row[11].getPart()
        # Go to the next row, get the fileid and geometry
        next_row = cur1.next()
        fileid2 = next_row[0]
        land2 = next_row[7]
        geom2 = next_row[11].getPart()
        # Create an Array
        array = arcpy.Array()
        # Test the parameters to see if it should be on-effort
        if ((fileid1 == fileid2) and (beaufort != None and beaufort <= 4) and ((visiblty != None and visiblty >= 2) or (oldvis == 1)) and \
                (land != 1 and land2 != 1) and (alt != None and alt <= 366) and \
                (not legtype == '0') and \
                (not (legtype == '7' and legstage == ' ')) and \
                (not (legtype == '9' and legstage == ' ')) and \
                (not (legtype == '5' and legstage == ' '))):
            array.add(geom1)
            array.add(geom2)
            polyline = arcpy.Polyline(array, sr)
            with arcpy.da.InsertCursor(outputFC, ['SHAPE@', "MONTH", "YEAR", "FILEID", "EVENTNO"]) as insertCursor1:
                insertCursor1.insertRow([polyline, month, year, fileid1, eventno])
            del insertCursor1
            array.removeAll()
        else:
            pass
del cur1
print("first with loop completed")
with arcpy.da.SearchCursor(fc_Sorted, fields) as cur2:
    cur2.next()
    for row in cur2:
        # Name variables and assign values starting on first record of table
        fileid1 = row[0]
        beaufort = row[1]
        legtype = row[2]
        legstage = row[3]
        visiblty = row[4]
        oldvis = row[5]
        alt = row[6]
        land = row[7]
        month = row[8]
        year = row[9]
        eventno = row[10]
        # Get the geometry of the point for that record
        geom1 = row[11].getPart()
        # Go to the next row, get the fileid and geometry
        next_row = cur2.next()
        fileid2 = next_row[0]
        land2 = next_row[7]
        geom2 = next_row[11].getPart()
        # Create an Array
        array = arcpy.Array()
        # Test the parameters to see if it should be on-effort
        if ((fileid1 == fileid2) and (beaufort != None and beaufort <= 4) and ((visiblty != None and visiblty >= 2) or (oldvis == 1)) and \
                (land != 1 and land2 != 1) and (alt != None and alt <= 366) and \
                (not legtype == '0') and \
                (not (legtype == '7' and legstage == ' ')) and \
                (not (legtype == '9' and legstage == ' ')) and \
                (not (legtype == '5' and legstage == ' '))):
            array.add(geom1)
            array.add(geom2)
            polyline = arcpy.Polyline(array, sr)
            with arcpy.da.InsertCursor(outputFC, ['SHAPE@', "MONTH", "YEAR", "FILEID", "EVENTNO"]) as insertCursor2:
                insertCursor2.insertRow([polyline, month, year, fileid1, eventno])
            del insertCursor2
            array.removeAll()
        else:
            pass
del cur2
print("Done")

This is the Error I get, but this changes depending on the run.  Sometimes it's in in insertCursor1 sometimes in insertCursor2 depending on how far it get's that run.

"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" "C:/Users/bhodge/Dropbox (New England Aquarium)/MonumentsWork/MonumentSpeciesDiversity/MonumentsPython/Trackline_Fields.py"
Traceback (most recent call last):
  File "C:/Users/bhodge/Dropbox (New England Aquarium)/MonumentsWork/MonumentSpeciesDiversity/MonumentsPython/Trackline_Fields.py", line 59, in <module>
    with arcpy.da.InsertCursor(outputFC, ['SHAPE@', "MONTH", "YEAR", "FILEID", "EVENTNO"]) as insertCursor1:
RuntimeError: cannot open 'C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb\EffortLines_TEST'

Process finished with exit code 1

Code description: Basically I'm running through a point table, looking at variables in the first row, and some in the next row and if those parameters meet my requirements, I draw a line to connect the points and save that to a featureclass I created along with some variables I need attached to those line segments.  I run through two searchCursors because of how I need to access the first and second line, it skips one iteration, (basically the first one compares points 1&2, 3&4, etc.) so I run it through a second searchCursor accessing the opposites (comparing 2&3, 4&5, etc.).  It's somehow locking the file I'm trying to write to in the insertCursor, but it does so randomly and I don't know why or how to fix it. Thanks in advance for any suggestions!

0 Kudos
14 Replies
BrookeHodge
New Contributor III

Thanks for this link.  I'm not sure it's helpful in my case since I'm not using the field calculator to populate my fields and am doing it with an insert cursor (and aren't doing any joins for summary statistics)... perhaps I can look into using an update cursor instead for the fields, but at the same time, I can accomplish populating the fields within the insert cursor which I need to use anyways to add the lines, and I don't believe that is where the issue lies.  

0 Kudos
DavidPike
MVP Frequent Contributor

I've not read through the code well enough but I will have a read through properly tomorrow.  You may want to access the geometry token (@SHAPE) in the search cursor and use that as an input to the geometry field of the insert cursor.  I'm also not sure what getpart does without googling when I have time as i'm far from expert.

Even without a dictionary you could have a list of lists which you iterate over

listy_mc_list_face = [    [field1, field2 … ,geometry object (@shape)], [field1, field2 ….], […    ]   ]

0 Kudos
BrookeHodge
New Contributor III

Hi,

Yep, I do use the @SHAPE token and do use that in the insert cursor (lines 25, then 59).  Here is was ESRI says the getpart does:

The getPart method returns an array of point objects for a particular part of the geometry if an index is specified. If an index is not specified, an array containing an array of point objects for each geometry part is returned.

I need to add the points to an array in which I create a line, so the getpart returns and array of the point I grab from the @SHAPE token.

As for your list of lists suggestion, I'm wondering.... If I iterate over a list, wouldn't that be doing what I"m doing now, just with a list instead of within each if statement?  I guess what I"m wondering is if when I iterate over the list, wouldn't it just be using the insertcursor over and over again as it iterates through, similar to what it's doing now?  Just wondering if I will have the same problem as I'm having now and it will randomly say it can't access the file when it's somewhere within the list iteration.

0 Kudos
CodyScott
Occasional Contributor

Out of curiosity, it looks like you're writing to a dropbox location based on the input/output. Have you tried writing to a folder that isn't a dropbox path?

Ignoring the suggestions above for now, does that work with your original code?

There might be some funny sync things happening between dropbox and the folder if you're writing a file continuously to it from python. 

Volume wise, if you're using a da.search cursor it shouldn't matter if it's 10 rows or 4.7 million in terms of memory usage. 

BrookeHodge
New Contributor III

Cody, you are a genius!  I should have thought of that sooner, but with 5+ years of working directly to dropbox with arc and never having a problem, I overlooked that.  You're were right!  The issue seems to be writing to the same file in dropbox over and over again, I wrote the file directly in my C drive and it worked.  

Thank you so much!

template | You Did It. The Crazy Son of a Bitch, You Did It | Know Your Meme