Arcpy for loop gives n-1 results

Nicole_Ueberschär · ‎02-10-2017

I have a problem with my for loop.

In an array profile_nrs all the numbers are stored I want to create points for in an existing feature class. When I print these values I get the expected nr of values (and also the correct values). Now I am creating points with an insertCursor and print in the end which point nr was added. I get all the (in this case) 18 points printed. But when I look into my file I receive only 17 points. Any advise is highly appreciated. I don't see any potential cause for this "misbehavior". My feeling tells me that it must not be looping through the numbers itself but through the index but then I wouldn't get nr 18 printed, would I?

Here the main part of the module:

for profile in profile_nrs:
    newPoint=arcpy.da.InsertCursor(points, fillnames)
    newPoint.insertRow(fillValues)
    print("point "+str(profile) +" added to All_points")‍‍‍‍‍‍‍‍

Inside the for loop I have a couple of other loops to get the fillnames and fillvalues but I don't see anywhere how this could influence the output. Especially if it says it added nr 18 but obviously it didn't...

Thanks in advance for helping me with this.

Nicole_Ueberschär · ‎02-13-2017

Thanks to both of your for taking your time to help me in this!

Dan, dedenting this whole part would mean moving it out of the profile_nrs loop, isn't it? But would it then still make a point for each of the profiles?

Joshua, in fact I had troubles exactly with this part. I didn't understand with the cursor, how to not copy the whole row but just a couple of attributes plus adding the shape somehow. I felt like having a knot in my head...

Using your snipplet gives me an "TypeError: can only concatenate list (not "tuple") to list" in your line 5 when "adding up" the fillValues.

I also tried already before with moving the insertCursor outside of the loop but it didn't make a difference in the result.

JoshuaBixby · ‎02-13-2017

My bad, I just roughed the code out without testing it. I made an update to Line 5, I believe it should work now.

Regarding moving the insert cursor, even if the end result isn't any different, you will want to move the insert cursor out of the loop for performance reasons. Unnecessarily recreating cursors within loops will add runtime to scripts, especially large loops.

RebeccaStrauch__GISP · ‎02-14-2017

I unmarked this thread as being answered since it appears it is still very active (sorry Dan Patterson )

n.ueberschaeresri-rw-esridist make sure to mark the correct answer once resolved, and to mark all those comments that were helpful.

DanPatterson_Retired · ‎02-14-2017

No sweat... no clue where it is going anyway

DanPatterson_Retired · ‎02-15-2017

perhaps you might want to mark this unanswered again

RichardFairhurst · ‎02-13-2017

As Dan, Joshua and others have pointed out, the creation of an InsertCursor should never happen within a loop. You only want to declare the InsertCursor once for any given table/fc (typically before any loops that involve it). Only the insertion of rows should be done within a loop:

points=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Campaigns\All_points"
pfields=arcpy.ListFields(points)
fillnames=[pfield.name for pfield in pfields if pfield.name != dsc.OIDFieldName]  #names for point feature class
newPoint=arcpy.da.InsertCursor(points, fillnames)‍
for profile in profile_nrs:
    # get a list of fillValues using some sort of method
    newPoint.insertRow(fillValues)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

RichardFairhurst · ‎02-13-2017

Your code is using far too many cursors. It is faster to blast through a complete table than it is to use expressions to limit the result most of the time, especially since you have to blast through the whole table to build your expression for the second cursor on that table. Don't bother trying to find the first records, just use it when going through the whole table and it meet your criteria and keep track of the fact you have already used it or break out of the loop. Anyway, I think the code below does what you are trying to do and is far less confusing.

I could make the code more efficient if I knew exactly what each input table contained. I did not assume that the table_name tables contained data for only one station. If that is actually the case then the second dictionary, the last for loop and the last two if clauses can be replaced with pfieldvalues = next(pCursor).

None of this may solve your original problem, since there are too many points of failure that could have happened in your prior script or in your input tables. Are you positive that all fields of the Test1 table correctly point to all of the tables created by your previous script? Did you verify your previous script created all of the expected tables? Are you sure there are no misspellings of any of the input table names or field values? If the script I wrote has the same failure as every other script you have tried I would say the cause of the failure is in the inputs to this script.

import arcpy
import os

#Workspace settings
arcpy.env.workspace=r"C:\data\2017\LakeKivu\Kivu_python.gdb"
arcpy.env.overwriteOutput= True

#input file
table = r"C:\data\2017\LakeKivu\Kivu_python.gdb\Test1"
points=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Campaigns\All_points"

dsc = arcpy.Describe(table)
fields = dsc.fields
# List all field names except the OID field
fieldnames = ["Station"] + [field.name for field in fields if field.name != dsc.OIDFieldName]

# create a dictionary of stations
stationDict = {}
with arcpy.da.SearchCursor(table, fieldnames) as cursor:
    for row in cursor:
        station = row[0]
        if not station in stationDict:
            stationDict[station] = str(row[1])+"_"+str(row[2])

pfieldnames=['Cruise','Station', 'Type', 'Date', 'Longitude', 'Latitude','UPI']

pfields=arcpy.ListFields(points)
fillnames=[pfield.name for pfield in pfields if pfield.name != dsc.OIDFieldName]  #names for point feature class

newPoint=arcpy.da.InsertCursor(points, fillnames)

stationpointDict = {}
for station in sorted(stationDict):
    table_name = stationDict[station]
    with arcpy.da.SearchCursor(table_name, pfieldnames) as pCursor:
        for pRow in pCursor:
            if station == pRow[1]:
                if not station in stationpointDict:
                    stationpointDict[station] = pRow
                    pfieldsvalues = pRow
                    ShapeVal = [pfieldsvalues[4], pfieldsvalues[5]]
                    fillValues = [ShapeVal] + list(pfieldsvalues)
                    print(fillValues) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
                    newPoint.insertRow(fillValues)
                    print("point " + str(station) + " added to All_points")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Nicole_Ueberschär · ‎02-13-2017

Thank you very much Richard for taking so much effort to work through (and obviously improve) my script!

I have to admit that I didn't take the time yet to read carefully through your blog posting Dan referred to before.

Actually, each table_name contains only the data for one station while the point feature class is supposed to contain all points taken from the input table(s). Additionally each table (each table_name as well as the corresponding point feature class) should receive an UID (the same as the table_name, incorrectly named UPI before) to be able to relate them later again.

At the moment the script doesn't write any points but I think I have to look through it again carefully... Thanks so far again! Will give you an update soon!

Nicole_Ueberschär · ‎02-13-2017

Ok, it works! Great job! Thanks a lot!

Just trying to get the differences now...

Why did you put the "Station" additionally as a field into fieldnames?

I don't have to delete the inputCursor in the end? Does this apply only for searchCursors (and UpdateCursors?) when NOT used with the "with"?

As I said in my last comment the table_name tables in fact only contain one station each. Those were already extracted in a previous script from a bigger file (as I tried to explain here: Nested Search/InsertCursors). The optimum would be to combine these two steps, so this will be my next attempt... I tried to replace as you said "the last for loop and the last two if clauses can be replaced with pfieldvalues = next(pCursor)" but I am still confused about what is happening where so I just left it for now. As long as it's working 😉

Thanks to all of you!

Nicole_Ueberschär · ‎02-15-2017

I'm trying now to combine my scripts but I fail to understand what you are doing Richard

Let me try to explain what I understand (or what I think to understand) and where I am struggling...

line 15:

fieldnames = ["Station"] + [field.name for field in fields if field.name != dsc.OIDFieldName]

plus line 19:

with arcpy.da.SearchCursor(table, fieldnames) as cursor:

Did you choose the additional field name "Station" by purpose because you saw in my pfieldnames that there is already a field called Station? Because, if I would choose a random name it would not find this column in my table when applying the SearchCursor, isn't it?

lines 18-23:

stationDict = {} with arcpy.da.SearchCursor(table, fieldnames) as cursor: for row in cursor: station = row[0] if not station in stationDict: stationDict[station] = str(row[1])+"_"+str(row[2])

This daSearchCursor goes through my input table including all fieldnames (including your defined "Station", excluding OID). The variable station gets the value of the attribute "Station" (because this is mentioned as the first field name). If the number does not exist yet the dictionary stationDict receives a new entry for the entry with the station x concatenating the values of the attributes 2 and 3 of the row of the current cursor. So, this dictionary contains only the concatenated name (the table_name) per row x. Right?

When I try now to follow your description which lines to replace I have instead of lines 32-45:

for station in sorted(stationDict):
    table_name = stationDict[station]
    with arcpy.da.SearchCursor(table_name, pfieldnames) as pCursor:
        pfieldsvalues = next(pCursor)‍‍‍‍
        ShapeVal = [pfieldsvalues[4], pfieldsvalues[5]]
        fillValues = [ShapeVal] + list(pfieldsvalues)
        print(fillValues)
        newPoint.insertRow(fillValues)
        print("point " + str(station) + " added to All_points")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This goes through my sorted stationDict, retrieves the table_name, and "opens" the table_name with my pfieldnames. Then it should write all values from the first row into pfieldsvalues, adds the Shape and the pfieldsvalues to the fillValues and inserts a new row into my point feature. Unfortunately it gives me the error "StopIteration" at my line 4. #

When I want to combine this part now with what I attempt to achieve with https://community.esri.com/thread/190087-nested-searchinsertcursors I get totally stranded. I will try to continue at this other posting to describe better what I want to do in consideration what I learned here.