topic Re: Arcpy for loop gives n-1 results in Python Questions

Arcpy for loop gives n-1 results

Nicole_Ueberschär — Sun, 12 Dec 2021 04:00:00 GMT

I have a problem with my for loop.

In an array profile_nrs all the numbers are stored I want to create points for in an existing feature class. When I print these values I get the expected nr of values (and also the correct values). Now I am creating points with an insertCursor and print in the end which point nr was added. I get all the (in this case) 18 points printed. But when I look into my file I receive only 17 points. Any advise is highly appreciated. I don't see any potential cause for this "misbehavior". My feeling tells me that it must not be looping through the numbers itself but through the index but then I wouldn't get nr 18 printed, would I?

Here the main part of the module:

for profile in profile_nrs:
    newPoint=arcpy.da.InsertCursor(points, fillnames)
    newPoint.insertRow(fillValues)
    print("point "+str(profile) +" added to All_points")‍‍‍‍‍‍‍‍

Inside the for loop I have a couple of other loops to get the fillnames and fillvalues but I don't see anywhere how this could influence the output. Especially if it says it added nr 18 but obviously it didn't...

Thanks in advance for helping me with this.

Re: Arcpy for loop gives n-1 results

DarrenWiens2 — Fri, 10 Feb 2017 22:00:14 GMT

Can you quickly post a screenshot of the attribute table showing the 17 points?

Re: Arcpy for loop gives n-1 results

DanPatterson_Retired — Fri, 10 Feb 2017 22:11:58 GMT

just in case looping through 18 objects would be the sequence 0 through 17 inclusive... for a total of 18

Re: Arcpy for loop gives n-1 results

JoshuaBixby — Fri, 10 Feb 2017 22:50:40 GMT

It helps to err on the side of giving more code than less. You have provided what you think is the functional and broken part of the code, but it could be the issue is upstream on how some of the variables are created/defined.

Also, just a comment on using an insert cursor, create it outside of the loop where you insert records.

Re: Arcpy for loop gives n-1 results

DanPatterson_Retired — Fri, 10 Feb 2017 23:30:51 GMT

I think this thread is related to this thread, which isn't closed

Re: Arcpy for loop gives n-1 results

Nicole_Ueberschär — Sat, 11 Feb 2017 17:59:19 GMT

Yes, correct. Didn't want to mix up the issues. I separated now both parts - creating first the tables and then the points but now I am stuck here...

Re: Arcpy for loop gives n-1 results

Nicole_Ueberschär — Sat, 11 Feb 2017 18:02:21 GMT

Here we go... (I hope it works with just copy-paste...)

Re: Arcpy for loop gives n-1 results

Nicole_Ueberschär — Sun, 12 Dec 2021 04:00:03 GMT

Didn't want to "overload" the code but here is the whole snipplet:

import arcpy
import os
#Workspace settings
arcpy.env.workspace=r"C:\data\2017\LakeKivu\Kivu_python.gdb"
arcpy.env.overwriteOutput= True

def unique_values(table, field):
    with arcpy.da.SearchCursor(table, [field]) as cursor:
        return sorted({row[0] for row in cursor})


#input file
table = r"C:\data\2017\LakeKivu\Kivu_python.gdb\Test1"

#Output files
DirOutProfiles=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Profiles"
DirOutCampaign=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Campaigns"
DirOutTables=r"C:\data\2017\LakeKivu\Kivu_python.gdb"
points=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Campaigns\All_points"

profile_nrs=unique_values(table, "Station")
print(profile_nrs)

dsc = arcpy.Describe(table)
fields = dsc.fields
# List all field names except the OID field
fieldnames = [field.name for field in fields if field.name != dsc.OIDFieldName]

pfields=arcpy.ListFields(points)
fillnames=[pfield.name for pfield in pfields if pfield.name != dsc.OIDFieldName]  #names for point feature class

# get tables
for profile in profile_nrs:
    expression="Station="+str(profile)
    print(expression)
    with arcpy.da.SearchCursor(table,fieldnames,expression) as sCursor:
        for sRow in sCursor:
            table_name=str(sRow[0])+"_"+str(sRow[1])
        del sRow


# Create point feature
    #Identify first OBJECTID used in new table
    oid_nrs=unique_values(table_name, "OBJECTID")
    FirstObject=oid_nrs[0]
    print(FirstObject)
    #Define expression to limit the extracted rows to the first row
    pexpression="OBJECTID="+str(oid_nrs[0])

    #Define field names used for point table
    pfieldnames=['Cruise','Station', 'Type', 'Date', 'Longitude', 'Latitude','UPI']
    print(pfieldnames)

    #get values from table table_name
    pfieldsvalues=[]

    for pname in pfieldnames:
        with arcpy.da.SearchCursor(table_name, pname, pexpression) as pCursor:
         for pRow in pCursor:
            fillvalue=pRow[0]
            pfieldsvalues.append(fillvalue)
    ShapeVal=[pfieldsvalues[4],pfieldsvalues[5]]
    print(fillnames)
    fillValues=[ShapeVal, pfieldsvalues[0],pfieldsvalues[1],pfieldsvalues[2],pfieldsvalues[3],pfieldsvalues[4],pfieldsvalues[5],pfieldsvalues[6],]
    print(fillValues)
    newPoint=arcpy.da.InsertCursor(points, fillnames)
    newPoint.insertRow(fillValues)
    station_nr=pfieldsvalues[1]
    print("point "+str(station_nr) +" added to All_points")
    del pRow
    del pCursor
del sCursor‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I'm pretty sure that there is a better/simpler way of creating my points but I didn't find it yet.

Maybe I miss somewhere a point in these Cursor methodologies. I couldn't figure out how to take only a certain number of attributes from the input table for my point feature PLUS adding somehow the shape values that's why it might look a bit irritating...

In short what I do here:

- profile_nrs takes all unique numbers in the attribute station (1-18) from input table Table1 (rows 21)

- I created in a different script tables based on these unique numbers. This way I got 18 tables with the according names (concatenating attributes 1 and 2, resulting into something like bla_1, bla_2,...) (rows 33-39)

- take only the first row of each table for creating the point (->pexpression, row 48+58)

- write in pfieldvalues the values from the input table (in this case one of the new 18 tables).(rows 54-61)

- fillnames includes the SHAPE attribute

- fillvalues takes the created SHAPE attribute value (ShapeVal) plus the values written in pfieldvalues. (row 64)

- then insert a row into points with fillvalues. (row 66/67).

Let's hope this makes it somehow clearer...

Re: Arcpy for loop gives n-1 results

DanPatterson_Retired — Sat, 11 Feb 2017 21:25:42 GMT

I am still having issues with the indentation, beginning a line 39 to the end... Is this a copy paste issue? it should be all dedented one level. There is still a whole load of cursor stuff going on

Re: Arcpy for loop gives n-1 results

JoshuaBixby — Sun, 12 Dec 2021 04:00:06 GMT

After taking some time to read over the whole code, I think I get the gist of what you are attempting to do, but I don't get the implementation. For example, I am really confused by lines 53 through 58. You appear to be creating multiple search cursors, each defined with one field and one row, just to fill a list of values that represent a subset of fields of that one row. Why not bypass looping over pfieldnames and just pass pfieldnames to the search cursor (replace lines 51-61):

with arcpy.da.SearchCursor(table_name, pfieldnames, pexpression) as pCursor:
    pfieldsvalues = next(pCursor)
ShapeVal = [pfieldsvalues[4], pfieldsvalues[5]]
print(fillnames)
fillValues = [ShapeVal] + list(pfieldsvalues)
print(fileValues)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Also, the creation of the insert cursor (line 66) can be moved outside of the for loop that starts at line 33.

Re: Arcpy for loop gives n-1 results

Nicole_Ueberschär — Mon, 13 Feb 2017 14:47:44 GMT

Thanks to both of your for taking your time to help me in this!

Dan, dedenting this whole part would mean moving it out of the profile_nrs loop, isn't it? But would it then still make a point for each of the profiles?

Joshua, in fact I had troubles exactly with this part. I didn't understand with the cursor, how to not copy the whole row but just a couple of attributes plus adding the shape somehow. I felt like having a knot in my head...

Using your snipplet gives me an "TypeError: can only concatenate list (not "tuple") to list" in your line 5 when "adding up" the fillValues.

I also tried already before with moving the insertCursor outside of the loop but it didn't make a difference in the result.

Re: Arcpy for loop gives n-1 results

JoshuaBixby — Mon, 13 Feb 2017 15:31:05 GMT

My bad, I just roughed the code out without testing it. I made an update to Line 5, I believe it should work now.

Regarding moving the insert cursor, even if the end result isn't any different, you will want to move the insert cursor out of the loop for performance reasons. Unnecessarily recreating cursors within loops will add runtime to scripts, especially large loops.

Re: Arcpy for loop gives n-1 results

RichardFairhurst — Sun, 12 Dec 2021 04:00:08 GMT

As Dan, Joshua and others have pointed out, the creation of an InsertCursor should never happen within a loop. You only want to declare the InsertCursor once for any given table/fc (typically before any loops that involve it). Only the insertion of rows should be done within a loop:

points=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Campaigns\All_points"
pfields=arcpy.ListFields(points)
fillnames=[pfield.name for pfield in pfields if pfield.name != dsc.OIDFieldName]  #names for point feature class
newPoint=arcpy.da.InsertCursor(points, fillnames)‍
for profile in profile_nrs:
    # get a list of fillValues using some sort of method
    newPoint.insertRow(fillValues)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Arcpy for loop gives n-1 results

RichardFairhurst — Sun, 12 Dec 2021 04:00:11 GMT

Your code is using far too many cursors. It is faster to blast through a complete table than it is to use expressions to limit the result most of the time, especially since you have to blast through the whole table to build your expression for the second cursor on that table. Don't bother trying to find the first records, just use it when going through the whole table and it meet your criteria and keep track of the fact you have already used it or break out of the loop. Anyway, I think the code below does what you are trying to do and is far less confusing.

I could make the code more efficient if I knew exactly what each input table contained. I did not assume that the table_name tables contained data for only one station. If that is actually the case then the second dictionary, the last for loop and the last two if clauses can be replaced with pfieldvalues = next(pCursor).

None of this may solve your original problem, since there are too many points of failure that could have happened in your prior script or in your input tables. Are you positive that all fields of the Test1 table correctly point to all of the tables created by your previous script? Did you verify your previous script created all of the expected tables? Are you sure there are no misspellings of any of the input table names or field values? If the script I wrote has the same failure as every other script you have tried I would say the cause of the failure is in the inputs to this script.

import arcpy
import os

#Workspace settings
arcpy.env.workspace=r"C:\data\2017\LakeKivu\Kivu_python.gdb"
arcpy.env.overwriteOutput= True

#input file
table = r"C:\data\2017\LakeKivu\Kivu_python.gdb\Test1"
points=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Campaigns\All_points"

dsc = arcpy.Describe(table)
fields = dsc.fields
# List all field names except the OID field
fieldnames = ["Station"] + [field.name for field in fields if field.name != dsc.OIDFieldName]

# create a dictionary of stations
stationDict = {}
with arcpy.da.SearchCursor(table, fieldnames) as cursor:
    for row in cursor:
        station = row[0]
        if not station in stationDict:
            stationDict[station] = str(row[1])+"_"+str(row[2])

pfieldnames=['Cruise','Station', 'Type', 'Date', 'Longitude', 'Latitude','UPI']

pfields=arcpy.ListFields(points)
fillnames=[pfield.name for pfield in pfields if pfield.name != dsc.OIDFieldName]  #names for point feature class

newPoint=arcpy.da.InsertCursor(points, fillnames)

stationpointDict = {}
for station in sorted(stationDict):
    table_name = stationDict[station]
    with arcpy.da.SearchCursor(table_name, pfieldnames) as pCursor:
        for pRow in pCursor:
            if station == pRow[1]:
                if not station in stationpointDict:
                    stationpointDict[station] = pRow
                    pfieldsvalues = pRow
                    ShapeVal = [pfieldsvalues[4], pfieldsvalues[5]]
                    fillValues = [ShapeVal] + list(pfieldsvalues)
                    print(fillValues) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
                    newPoint.insertRow(fillValues)
                    print("point " + str(station) + " added to All_points")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Arcpy for loop gives n-1 results

Nicole_Ueberschär — Mon, 13 Feb 2017 19:57:51 GMT

Thank you very much Richard for taking so much effort to work through (and obviously improve) my script!

I have to admit that I didn't take the time yet to read carefully through your blog posting Dan referred to before.

Actually, each table_name contains only the data for one station while the point feature class is supposed to contain all points taken from the input table(s). Additionally each table (each table_name as well as the corresponding point feature class) should receive an UID (the same as the table_name, incorrectly named UPI before) to be able to relate them later again.

At the moment the script doesn't write any points but I think I have to look through it again carefully... Thanks so far again! Will give you an update soon!

Re: Arcpy for loop gives n-1 results

Nicole_Ueberschär — Tue, 14 Feb 2017 05:54:32 GMT

Ok, it works! Great job! Thanks a lot!

Just trying to get the differences now...

Why did you put the "Station" additionally as a field into fieldnames?

I don't have to delete the inputCursor in the end? Does this apply only for searchCursors (and UpdateCursors?) when NOT used with the "with"?

As I said in my last comment the table_name tables in fact only contain one station each. Those were already extracted in a previous script from a bigger file (as I tried to explain here: Nested Search/InsertCursors). The optimum would be to combine these two steps, so this will be my next attempt... I tried to replace as you said "the last for loop and the last two if clauses can be replaced with pfieldvalues = next(pCursor)" but I am still confused about what is happening where so I just left it for now. As long as it's working 😉

Thanks to all of you!

Re: Arcpy for loop gives n-1 results

RebeccaStrauch__GISP — Tue, 14 Feb 2017 15:51:13 GMT

I unmarked this thread as being answered since it appears it is still very active (sorry Dan Patterson )

n.ueberschaeresri-rw-esridist make sure to mark the correct answer once resolved, and to mark all those comments that were helpful.

Re: Arcpy for loop gives n-1 results

DanPatterson_Retired — Tue, 14 Feb 2017 17:58:59 GMT

No sweat... no clue where it is going anyway

Re: Arcpy for loop gives n-1 results

Nicole_Ueberschär — Sun, 12 Dec 2021 04:00:14 GMT

I'm trying now to combine my scripts but I fail to understand what you are doing Richard

Let me try to explain what I understand (or what I think to understand) and where I am struggling...

line 15:

fieldnames = ["Station"] + [field.name for field in fields if field.name != dsc.OIDFieldName]

plus line 19:

with arcpy.da.SearchCursor(table, fieldnames) as cursor:

Did you choose the additional field name "Station" by purpose because you saw in my pfieldnames that there is already a field called Station? Because, if I would choose a random name it would not find this column in my table when applying the SearchCursor, isn't it?

lines 18-23:

stationDict = {} with arcpy.da.SearchCursor(table, fieldnames) as cursor: for row in cursor: station = row[0] if not station in stationDict: stationDict[station] = str(row[1])+"_"+str(row[2])

This daSearchCursor goes through my input table including all fieldnames (including your defined "Station", excluding OID). The variable station gets the value of the attribute "Station" (because this is mentioned as the first field name). If the number does not exist yet the dictionary stationDict receives a new entry for the entry with the station x concatenating the values of the attributes 2 and 3 of the row of the current cursor. So, this dictionary contains only the concatenated name (the table_name) per row x. Right?

When I try now to follow your description which lines to replace I have instead of lines 32-45:

for station in sorted(stationDict):
    table_name = stationDict[station]
    with arcpy.da.SearchCursor(table_name, pfieldnames) as pCursor:
        pfieldsvalues = next(pCursor)‍‍‍‍
        ShapeVal = [pfieldsvalues[4], pfieldsvalues[5]]
        fillValues = [ShapeVal] + list(pfieldsvalues)
        print(fillValues)
        newPoint.insertRow(fillValues)
        print("point " + str(station) + " added to All_points")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This goes through my sorted stationDict, retrieves the table_name, and "opens" the table_name with my pfieldnames. Then it should write all values from the first row into pfieldsvalues, adds the Shape and the pfieldsvalues to the fillValues and inserts a new row into my point feature. Unfortunately it gives me the error "StopIteration" at my line 4. #

When I want to combine this part now with what I attempt to achieve with https://community.esri.com/thread/190087-nested-searchinsertcursors I get totally stranded. I will try to continue at this other posting to describe better what I want to do in consideration what I learned here.

Re: Arcpy for loop gives n-1 results

RichardFairhurst — Sun, 12 Dec 2021 04:00:16 GMT

Nicole:

Line 15 adds the station field as the first field in the list, because this is field holding the unique dictionary key value used match records between each table per your description of what you wanted to do. The field list includes the station twice, once for the dictionary key and once for the value of the field for a given row, which is important later when the station field has to be in the position it exists relative to other fields to hold all the values of a row object that is passed to the insert cursor.

Line 19 reads the records of the search table that contains the table names.

Lines 18-23 - you are correct that the dictionary is adding each station only the first time it is encountered to the dictionary and the key value is the concatenation of the two fields that make up your table names, per your description of what you had set up.

For the modification of the code, I did not test using the next method. Go back to using the for loop and break out of it once you have created a feature for the table. I.e.:

for station in sorted(stationDict):
    table_name = stationDict[station]
    with arcpy.da.SearchCursor(table_name, pfieldnames) as pCursor:
        for pRow in pCursor:
            stationpointDict[station] = pRow
            pfieldsvalues = pRow
            ShapeVal = [pfieldsvalues[4], pfieldsvalues[5]]
            fillValues = [ShapeVal] + list(pfieldsvalues)
            print(fillValues) 
            newPoint.insertRow(fillValues)
            print("point " + str(station) + " added to All_points")
            break
   del pCursor‍‍‍‍‍‍‍‍‍‍‍‍‍