cannot open file using arcpy.da.InsertCursor (randomly)

[BrookeHodge]

Hi all, I'm hoping someone can help me a problem I can't figure out.  Below is my code.  I keep getting a RuntimeError: cannot open 'file I'm writing to' using a arcpy.da.InsertCursor.  It keeps locking for some reason, I do not have arc open when running (running it from PyCharm), and it will write to the file thousands of times... I am writing line segments to a feature class and it will do it thousands of times, but then seemingly randomly say it can't access it.  I have run it many times and it get's locked up in different places each time.  The only info I can find is that people need to close their cursors, but all my cursors (both search and insert cursors) are closed after each use.  I'm not sure what else to do.  Any ideas?  Description of what code is trying to do below if that is helpful at all as well as a screen shot of the error.

import arcpy

# Set environments / workspace.  Workspace should point to a gdb that has all the effort point feature classes
arcpy.env.workspace = r"C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb"
arcpy.env.overwriteOutput = True
# This is the input points
fc = r"C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb\EffortPoints_Prj_TEST_B"
# This is the output lines
sr = arcpy.Describe(fc).spatialReference
outputFC = arcpy.management.CreateFeatureclass(r"C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb", "EffortLines_TEST", "POLYLINE", "", "DISABLED", "DISABLED", sr)
# Add desired fields to outputFC
arcpy.AddField_management(outputFC, "MONTH", "TEXT")
arcpy.AddField_management(outputFC, "YEAR", "SHORT")
arcpy.AddField_management(outputFC, "FILEID", "TEXT")
arcpy.AddField_management(outputFC, "EVENTNO", "LONG")

# Make a feature Layer of the input points to work off.  This layer is temporary and will not persist after session ends unless saved
arcpy.MakeFeatureLayer_management(fc, "fc_lyr")

# Sort the table by fileid and eventno and create a new fc called fc_sort
sort_fields = [["FILEID", "ASCENDING"], ["EVENTNO", "ASCENDING"]]
fc_Sorted = arcpy.Sort_management("fc_lyr", "fc_sort", sort_fields)

# Create the search cursor
fields = ['FILEID', 'beauf_num', 'LEGTYPE', 'LEGSTAGE', 'vis_num', 'olviz_num', 'alt_num', 'LAND', 'MONTH', 'YEAR', "EVENTNO", "SHAPE@"]
with arcpy.da.SearchCursor(fc_Sorted, fields) as cur1:
    for row in cur1:
        # Name variables and assign values starting on first record of table
        fileid1 = row[0]
        beaufort = row[1]
        legtype = row[2]
        legstage = row[3]
        visiblty = row[4]
        oldvis = row[5]
        alt = row[6]
        land = row[7]
        month = row[8]
        year = row[9]
        eventno = row[10]
        # Get the geometry of the point for that record
        geom1 = row[11].getPart()
        # Go to the next row, get the fileid and geometry
        next_row = cur1.next()
        fileid2 = next_row[0]
        land2 = next_row[7]
        geom2 = next_row[11].getPart()
        # Create an Array
        array = arcpy.Array()
        # Test the parameters to see if it should be on-effort
        if ((fileid1 == fileid2) and (beaufort != None and beaufort <= 4) and ((visiblty != None and visiblty >= 2) or (oldvis == 1)) and \
                (land != 1 and land2 != 1) and (alt != None and alt <= 366) and \
                (not legtype == '0') and \
                (not (legtype == '7' and legstage == ' ')) and \
                (not (legtype == '9' and legstage == ' ')) and \
                (not (legtype == '5' and legstage == ' '))):
            array.add(geom1)
            array.add(geom2)
            polyline = arcpy.Polyline(array, sr)
            with arcpy.da.InsertCursor(outputFC, ['SHAPE@', "MONTH", "YEAR", "FILEID", "EVENTNO"]) as insertCursor1:
                insertCursor1.insertRow([polyline, month, year, fileid1, eventno])
            del insertCursor1
            array.removeAll()
        else:
            pass
del cur1
print("first with loop completed")
with arcpy.da.SearchCursor(fc_Sorted, fields) as cur2:
    cur2.next()
    for row in cur2:
        # Name variables and assign values starting on first record of table
        fileid1 = row[0]
        beaufort = row[1]
        legtype = row[2]
        legstage = row[3]
        visiblty = row[4]
        oldvis = row[5]
        alt = row[6]
        land = row[7]
        month = row[8]
        year = row[9]
        eventno = row[10]
        # Get the geometry of the point for that record
        geom1 = row[11].getPart()
        # Go to the next row, get the fileid and geometry
        next_row = cur2.next()
        fileid2 = next_row[0]
        land2 = next_row[7]
        geom2 = next_row[11].getPart()
        # Create an Array
        array = arcpy.Array()
        # Test the parameters to see if it should be on-effort
        if ((fileid1 == fileid2) and (beaufort != None and beaufort <= 4) and ((visiblty != None and visiblty >= 2) or (oldvis == 1)) and \
                (land != 1 and land2 != 1) and (alt != None and alt <= 366) and \
                (not legtype == '0') and \
                (not (legtype == '7' and legstage == ' ')) and \
                (not (legtype == '9' and legstage == ' ')) and \
                (not (legtype == '5' and legstage == ' '))):
            array.add(geom1)
            array.add(geom2)
            polyline = arcpy.Polyline(array, sr)
            with arcpy.da.InsertCursor(outputFC, ['SHAPE@', "MONTH", "YEAR", "FILEID", "EVENTNO"]) as insertCursor2:
                insertCursor2.insertRow([polyline, month, year, fileid1, eventno])
            del insertCursor2
            array.removeAll()
        else:
            pass
del cur2
print("Done")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This is the Error I get, but this changes depending on the run.  Sometimes it's in in insertCursor1 sometimes in insertCursor2 depending on how far it get's that run.

"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" "C:/Users/bhodge/Dropbox (New England Aquarium)/MonumentsWork/MonumentSpeciesDiversity/MonumentsPython/Trackline_Fields.py"
Traceback (most recent call last):
  File "C:/Users/bhodge/Dropbox (New England Aquarium)/MonumentsWork/MonumentSpeciesDiversity/MonumentsPython/Trackline_Fields.py", line 59, in <module>
    with arcpy.da.InsertCursor(outputFC, ['SHAPE@', "MONTH", "YEAR", "FILEID", "EVENTNO"]) as insertCursor1:
RuntimeError: cannot open 'C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb\EffortLines_TEST'

Process finished with exit code 1‍‍‍‍‍‍‍

Code description: Basically I'm running through a point table, looking at variables in the first row, and some in the next row and if those parameters meet my requirements, I draw a line to connect the points and save that to a featureclass I created along with some variables I need attached to those line segments.  I run through two searchCursors because of how I need to access the first and second line, it skips one iteration, (basically the first one compares points 1&2, 3&4, etc.) so I run it through a second searchCursor accessing the opposites (comparing 2&3, 4&5, etc.).  It's somehow locking the file I'm trying to write to in the insertCursor, but it does so randomly and I don't know why or how to fix it. Thanks in advance for any suggestions!

[DavidPike]

Can you just store the results of the Search Cursor in a dictionary then throw them in at the end with an Insert Cursor instead?

[BrookeHodge]

Thanks for  your response.  I was thinking if that could work.  I've never used a dictionary before but was trying to figure out it it could hold both the geometry of the line as well as all the attributes for the fields I want to populate per record in a way I could then write to a featurclass all at once at the end.  That I haven't figured out yet.  Do you know if that is possible with a dictionary?  Basically I need to create the line from two points, save that line geometry, and grab variable values from the points from a number of fields and store those with the line, then be able to write all of that to a new feature class.  Previously when I was working with this data, I was just creating lines and did not need to keep attribute information from the original point data so I just stored all the lines in a list and wrote them all to a featureclass at the end.  That worked, but I now need to keep attribute values and associate them back with the lines so that's where I'm struggling the best method to accomplish that.  I was reading up on dictionaries today, but couldn't figure out how to accomplish this with them, however, maybe I just haven't found the right information/example?  If you know of any examples or resources for this I would be grateful!

[JoeBorgione]

See :  /blogs/richard_fairhurst/2014/11/08/turbo-charging-data-manipulation-with-python-cursors-and-diction... 

[JoeBorgione]

Let me see if I have this right: you (re)create a feature class (outputFC) and add fields to it.  Then you open an insert cursor on that same outputFC, and that's when it fails?  Like it's getting hung up because of the add field steps?

[BrookeHodge]

Not quite.  You have the beginning right.  I create a feature class and add fields to it.  I then create lines from points as well as grab variable values to populate the fields.  I use an insert cursor on the created feature class to add the created line and populate the fields. Then it goes to the next record and continues.  It successfully does this a number of times (it will write 100s to 1000s of lines and populate fields to this feature class).  Then seemingly randomly, it will then say it cannot access the feature class (even thought it had just accessed it 100s to 1000s of times).  Each time I run, it stops at a different point.  For instance, one time it will write 478 lines to the feature class, then throw the error.  Next time I run it it will do 6,893 lines, the next time 2,645 lines.  It seem like it's getting locked up somehow, but I don't know why.  I am sure to release all my cursors after you and have it open no where else when running.  

[JoeBorgione]

I wonder if it's a memory issue; just a WAG (wild a$$ guess)

[BrookeHodge]

I watched my memory while it was running, and never got above like 32%.  My CPU hit 100 a couple times, but it didn't throw the error then and continued.  I ran it again this morning and it magically worked (I changed nothing) once, but then I tried it with more data (I was testing with about 5,000 points), but then just tried it with about 400,000 points (I actually have 4.7  million I need to run, but I think I'll need to break it up), and it failed again with the same error that it could not open the file... sigh.

[JoeBorgione]

When in doubt I always defer to these guys...  Seriously though, I'm at a total loss...

[BrookeHodge]

Thanks Joe!  I appreciate your time and effort:.)