topic Re: cannot open file using arcpy.da.InsertCursor (randomly) in Python Questions

cannot open file using arcpy.da.InsertCursor (randomly)

BrookeHodge — Sun, 12 Dec 2021 00:05:01 GMT

Hi all, I'm hoping someone can help me a problem I can't figure out. Below is my code. I keep getting a RuntimeError: cannot open 'file I'm writing to' using a arcpy.da.InsertCursor. It keeps locking for some reason, I do not have arc open when running (running it from PyCharm), and it will write to the file thousands of times... I am writing line segments to a feature class and it will do it thousands of times, but then seemingly randomly say it can't access it. I have run it many times and it get's locked up in different places each time. The only info I can find is that people need to close their cursors, but all my cursors (both search and insert cursors) are closed after each use. I'm not sure what else to do. Any ideas? Description of what code is trying to do below if that is helpful at all as well as a screen shot of the error.

import arcpy

# Set environments / workspace.  Workspace should point to a gdb that has all the effort point feature classes
arcpy.env.workspace = r"C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb"
arcpy.env.overwriteOutput = True
# This is the input points
fc = r"C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb\EffortPoints_Prj_TEST_B"
# This is the output lines
sr = arcpy.Describe(fc).spatialReference
outputFC = arcpy.management.CreateFeatureclass(r"C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb", "EffortLines_TEST", "POLYLINE", "", "DISABLED", "DISABLED", sr)
# Add desired fields to outputFC
arcpy.AddField_management(outputFC, "MONTH", "TEXT")
arcpy.AddField_management(outputFC, "YEAR", "SHORT")
arcpy.AddField_management(outputFC, "FILEID", "TEXT")
arcpy.AddField_management(outputFC, "EVENTNO", "LONG")

# Make a feature Layer of the input points to work off.  This layer is temporary and will not persist after session ends unless saved
arcpy.MakeFeatureLayer_management(fc, "fc_lyr")

# Sort the table by fileid and eventno and create a new fc called fc_sort
sort_fields = [["FILEID", "ASCENDING"], ["EVENTNO", "ASCENDING"]]
fc_Sorted = arcpy.Sort_management("fc_lyr", "fc_sort", sort_fields)

# Create the search cursor
fields = ['FILEID', 'beauf_num', 'LEGTYPE', 'LEGSTAGE', 'vis_num', 'olviz_num', 'alt_num', 'LAND', 'MONTH', 'YEAR', "EVENTNO", "SHAPE@"]
with arcpy.da.SearchCursor(fc_Sorted, fields) as cur1:
    for row in cur1:
        # Name variables and assign values starting on first record of table
        fileid1 = row[0]
        beaufort = row[1]
        legtype = row[2]
        legstage = row[3]
        visiblty = row[4]
        oldvis = row[5]
        alt = row[6]
        land = row[7]
        month = row[8]
        year = row[9]
        eventno = row[10]
        # Get the geometry of the point for that record
        geom1 = row[11].getPart()
        # Go to the next row, get the fileid and geometry
        next_row = cur1.next()
        fileid2 = next_row[0]
        land2 = next_row[7]
        geom2 = next_row[11].getPart()
        # Create an Array
        array = arcpy.Array()
        # Test the parameters to see if it should be on-effort
        if ((fileid1 == fileid2) and (beaufort != None and beaufort <= 4) and ((visiblty != None and visiblty >= 2) or (oldvis == 1)) and \
                (land != 1 and land2 != 1) and (alt != None and alt <= 366) and \
                (not legtype == '0') and \
                (not (legtype == '7' and legstage == ' ')) and \
                (not (legtype == '9' and legstage == ' ')) and \
                (not (legtype == '5' and legstage == ' '))):
            array.add(geom1)
            array.add(geom2)
            polyline = arcpy.Polyline(array, sr)
            with arcpy.da.InsertCursor(outputFC, ['SHAPE@', "MONTH", "YEAR", "FILEID", "EVENTNO"]) as insertCursor1:
                insertCursor1.insertRow([polyline, month, year, fileid1, eventno])
            del insertCursor1
            array.removeAll()
        else:
            pass
del cur1
print("first with loop completed")
with arcpy.da.SearchCursor(fc_Sorted, fields) as cur2:
    cur2.next()
    for row in cur2:
        # Name variables and assign values starting on first record of table
        fileid1 = row[0]
        beaufort = row[1]
        legtype = row[2]
        legstage = row[3]
        visiblty = row[4]
        oldvis = row[5]
        alt = row[6]
        land = row[7]
        month = row[8]
        year = row[9]
        eventno = row[10]
        # Get the geometry of the point for that record
        geom1 = row[11].getPart()
        # Go to the next row, get the fileid and geometry
        next_row = cur2.next()
        fileid2 = next_row[0]
        land2 = next_row[7]
        geom2 = next_row[11].getPart()
        # Create an Array
        array = arcpy.Array()
        # Test the parameters to see if it should be on-effort
        if ((fileid1 == fileid2) and (beaufort != None and beaufort <= 4) and ((visiblty != None and visiblty >= 2) or (oldvis == 1)) and \
                (land != 1 and land2 != 1) and (alt != None and alt <= 366) and \
                (not legtype == '0') and \
                (not (legtype == '7' and legstage == ' ')) and \
                (not (legtype == '9' and legstage == ' ')) and \
                (not (legtype == '5' and legstage == ' '))):
            array.add(geom1)
            array.add(geom2)
            polyline = arcpy.Polyline(array, sr)
            with arcpy.da.InsertCursor(outputFC, ['SHAPE@', "MONTH", "YEAR", "FILEID", "EVENTNO"]) as insertCursor2:
                insertCursor2.insertRow([polyline, month, year, fileid1, eventno])
            del insertCursor2
            array.removeAll()
        else:
            pass
del cur2
print("Done")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This is the Error I get, but this changes depending on the run. Sometimes it's in in insertCursor1 sometimes in insertCursor2 depending on how far it get's that run.

"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" "C:/Users/bhodge/Dropbox (New England Aquarium)/MonumentsWork/MonumentSpeciesDiversity/MonumentsPython/Trackline_Fields.py"
Traceback (most recent call last):
  File "C:/Users/bhodge/Dropbox (New England Aquarium)/MonumentsWork/MonumentSpeciesDiversity/MonumentsPython/Trackline_Fields.py", line 59, in <module>
    with arcpy.da.InsertCursor(outputFC, ['SHAPE@', "MONTH", "YEAR", "FILEID", "EVENTNO"]) as insertCursor1:
RuntimeError: cannot open 'C:\Users\bhodge\Dropbox (New England Aquarium)\MonumentsWork\MonumentSpeciesDiversity\DataFromBob\DerivedData.gdb\EffortLines_TEST'

Process finished with exit code 1‍‍‍‍‍‍‍

Code description: Basically I'm running through a point table, looking at variables in the first row, and some in the next row and if those parameters meet my requirements, I draw a line to connect the points and save that to a featureclass I created along with some variables I need attached to those line segments. I run through two searchCursors because of how I need to access the first and second line, it skips one iteration, (basically the first one compares points 1&2, 3&4, etc.) so I run it through a second searchCursor accessing the opposites (comparing 2&3, 4&5, etc.). It's somehow locking the file I'm trying to write to in the insertCursor, but it does so randomly and I don't know why or how to fix it. Thanks in advance for any suggestions!

Re: cannot open file using arcpy.da.InsertCursor (randomly)

DavidPike — Thu, 17 Sep 2020 20:16:22 GMT

Can you just store the results of the Search Cursor in a dictionary then throw them in at the end with an Insert Cursor instead?

Re: cannot open file using arcpy.da.InsertCursor (randomly)

JoeBorgione — Thu, 17 Sep 2020 20:57:05 GMT

Let me see if I have this right: you (re)create a feature class (outputFC) and add fields to it. Then you open an insert cursor on that same outputFC, and that's when it fails? Like it's getting hung up because of the add field steps?

Re: cannot open file using arcpy.da.InsertCursor (randomly)

BrookeHodge — Thu, 17 Sep 2020 21:29:41 GMT

Thanks for your response. I was thinking if that could work. I've never used a dictionary before but was trying to figure out it it could hold both the geometry of the line as well as all the attributes for the fields I want to populate per record in a way I could then write to a featurclass all at once at the end. That I haven't figured out yet. Do you know if that is possible with a dictionary? Basically I need to create the line from two points, save that line geometry, and grab variable values from the points from a number of fields and store those with the line, then be able to write all of that to a new feature class. Previously when I was working with this data, I was just creating lines and did not need to keep attribute information from the original point data so I just stored all the lines in a list and wrote them all to a featureclass at the end. That worked, but I now need to keep attribute values and associate them back with the lines so that's where I'm struggling the best method to accomplish that. I was reading up on dictionaries today, but couldn't figure out how to accomplish this with them, however, maybe I just haven't found the right information/example? If you know of any examples or resources for this I would be grateful!

Re: cannot open file using arcpy.da.InsertCursor (randomly)

JoeBorgione — Thu, 17 Sep 2020 21:31:57 GMT

See : /blogs/richard_fairhurst/2014/11/08/turbo-charging-data-manipulation-with-python-cursors-and-dictionaries

Re: cannot open file using arcpy.da.InsertCursor (randomly)

BrookeHodge — Thu, 17 Sep 2020 21:36:09 GMT

Not quite. You have the beginning right. I create a feature class and add fields to it. I then create lines from points as well as grab variable values to populate the fields. I use an insert cursor on the created feature class to add the created line and populate the fields. Then it goes to the next record and continues. It successfully does this a number of times (it will write 100s to 1000s of lines and populate fields to this feature class). Then seemingly randomly, it will then say it cannot access the feature class (even thought it had just accessed it 100s to 1000s of times). Each time I run, it stops at a different point. For instance, one time it will write 478 lines to the feature class, then throw the error. Next time I run it it will do 6,893 lines, the next time 2,645 lines. It seem like it's getting locked up somehow, but I don't know why. I am sure to release all my cursors after you and have it open no where else when running.

Re: cannot open file using arcpy.da.InsertCursor (randomly)

BrookeHodge — Thu, 17 Sep 2020 21:44:03 GMT

Thanks for this link. I'm not sure it's helpful in my case since I'm not using the field calculator to populate my fields and am doing it with an insert cursor (and aren't doing any joins for summary statistics)... perhaps I can look into using an update cursor instead for the fields, but at the same time, I can accomplish populating the fields within the insert cursor which I need to use anyways to add the lines, and I don't believe that is where the issue lies.

Re: cannot open file using arcpy.da.InsertCursor (randomly)

DavidPike — Thu, 17 Sep 2020 21:58:40 GMT

I've not read through the code well enough but I will have a read through properly tomorrow. You may want to access the geometry token (@SHAPE) in the search cursor and use that as an input to the geometry field of the insert cursor. I'm also not sure what getpart does without googling when I have time as i'm far from expert.

Even without a dictionary you could have a list of lists which you iterate over

listy_mc_list_face = [ [field1, field2 … ,geometry object (@shape)], [field1, field2 ….], [… ] ]

Re: cannot open file using arcpy.da.InsertCursor (randomly)

JoeBorgione — Thu, 17 Sep 2020 22:13:10 GMT

I wonder if it's a memory issue; just a WAG (wild a$$ guess)

Re: cannot open file using arcpy.da.InsertCursor (randomly)

BrookeHodge — Fri, 18 Sep 2020 14:35:15 GMT

I watched my memory while it was running, and never got above like 32%. My CPU hit 100 a couple times, but it didn't throw the error then and continued. I ran it again this morning and it magically worked (I changed nothing) once, but then I tried it with more data (I was testing with about 5,000 points), but then just tried it with about 400,000 points (I actually have 4.7 million I need to run, but I think I'll need to break it up), and it failed again with the same error that it could not open the file... sigh.

Re: cannot open file using arcpy.da.InsertCursor (randomly)

BrookeHodge — Fri, 18 Sep 2020 14:46:23 GMT

Hi,

Yep, I do use the @SHAPE token and do use that in the insert cursor (lines 25, then 59). Here is was ESRI says the getpart does:

The getPart method returns an array of point objects for a particular part of the geometry if an index is specified. If an index is not specified, an array containing an array of point objects for each geometry part is returned.

I need to add the points to an array in which I create a line, so the getpart returns and array of the point I grab from the @SHAPE token.

As for your list of lists suggestion, I'm wondering.... If I iterate over a list, wouldn't that be doing what I"m doing now, just with a list instead of within each if statement? I guess what I"m wondering is if when I iterate over the list, wouldn't it just be using the insertcursor over and over again as it iterates through, similar to what it's doing now? Just wondering if I will have the same problem as I'm having now and it will randomly say it can't access the file when it's somewhere within the list iteration.

Re: cannot open file using arcpy.da.InsertCursor (randomly)

JoeBorgione — Fri, 18 Sep 2020 14:46:32 GMT

When in doubt I always defer to these guys... Seriously though, I'm at a total loss...

Re: cannot open file using arcpy.da.InsertCursor (randomly)

BrookeHodge — Fri, 18 Sep 2020 14:49:00 GMT

Thanks Joe! I appreciate your time and effort:.)

Re: cannot open file using arcpy.da.InsertCursor (randomly)

CodyScott — Mon, 28 Sep 2020 15:37:10 GMT

Out of curiosity, it looks like you're writing to a dropbox location based on the input/output. Have you tried writing to a folder that isn't a dropbox path?

Ignoring the suggestions above for now, does that work with your original code?

There might be some funny sync things happening between dropbox and the folder if you're writing a file continuously to it from python.

Volume wise, if you're using a da.search cursor it shouldn't matter if it's 10 rows or 4.7 million in terms of memory usage.

Re: cannot open file using arcpy.da.InsertCursor (randomly)

BrookeHodge — Tue, 29 Sep 2020 18:12:38 GMT

Cody, you are a genius! I should have thought of that sooner, but with 5+ years of working directly to dropbox with arc and never having a problem, I overlooked that. You're were right! The issue seems to be writing to the same file in dropbox over and over again, I wrote the file directly in my C drive and it worked.

Thank you so much!