Arcpy for loop gives n-1 results

Nicole_Ueberschär · ‎02-10-2017

I have a problem with my for loop.

In an array profile_nrs all the numbers are stored I want to create points for in an existing feature class. When I print these values I get the expected nr of values (and also the correct values). Now I am creating points with an insertCursor and print in the end which point nr was added. I get all the (in this case) 18 points printed. But when I look into my file I receive only 17 points. Any advise is highly appreciated. I don't see any potential cause for this "misbehavior". My feeling tells me that it must not be looping through the numbers itself but through the index but then I wouldn't get nr 18 printed, would I?

Here the main part of the module:

for profile in profile_nrs:
    newPoint=arcpy.da.InsertCursor(points, fillnames)
    newPoint.insertRow(fillValues)
    print("point "+str(profile) +" added to All_points")‍‍‍‍‍‍‍‍

Inside the for loop I have a couple of other loops to get the fillnames and fillvalues but I don't see anywhere how this could influence the output. Especially if it says it added nr 18 but obviously it didn't...

Thanks in advance for helping me with this.

RichardFairhurst · ‎02-13-2017

Your code is using far too many cursors. It is faster to blast through a complete table than it is to use expressions to limit the result most of the time, especially since you have to blast through the whole table to build your expression for the second cursor on that table. Don't bother trying to find the first records, just use it when going through the whole table and it meet your criteria and keep track of the fact you have already used it or break out of the loop. Anyway, I think the code below does what you are trying to do and is far less confusing.

I could make the code more efficient if I knew exactly what each input table contained. I did not assume that the table_name tables contained data for only one station. If that is actually the case then the second dictionary, the last for loop and the last two if clauses can be replaced with pfieldvalues = next(pCursor).

None of this may solve your original problem, since there are too many points of failure that could have happened in your prior script or in your input tables. Are you positive that all fields of the Test1 table correctly point to all of the tables created by your previous script? Did you verify your previous script created all of the expected tables? Are you sure there are no misspellings of any of the input table names or field values? If the script I wrote has the same failure as every other script you have tried I would say the cause of the failure is in the inputs to this script.

import arcpy
import os

#Workspace settings
arcpy.env.workspace=r"C:\data\2017\LakeKivu\Kivu_python.gdb"
arcpy.env.overwriteOutput= True

#input file
table = r"C:\data\2017\LakeKivu\Kivu_python.gdb\Test1"
points=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Campaigns\All_points"

dsc = arcpy.Describe(table)
fields = dsc.fields
# List all field names except the OID field
fieldnames = ["Station"] + [field.name for field in fields if field.name != dsc.OIDFieldName]

# create a dictionary of stations
stationDict = {}
with arcpy.da.SearchCursor(table, fieldnames) as cursor:
    for row in cursor:
        station = row[0]
        if not station in stationDict:
            stationDict[station] = str(row[1])+"_"+str(row[2])

pfieldnames=['Cruise','Station', 'Type', 'Date', 'Longitude', 'Latitude','UPI']

pfields=arcpy.ListFields(points)
fillnames=[pfield.name for pfield in pfields if pfield.name != dsc.OIDFieldName]  #names for point feature class

newPoint=arcpy.da.InsertCursor(points, fillnames)

stationpointDict = {}
for station in sorted(stationDict):
    table_name = stationDict[station]
    with arcpy.da.SearchCursor(table_name, pfieldnames) as pCursor:
        for pRow in pCursor:
            if station == pRow[1]:
                if not station in stationpointDict:
                    stationpointDict[station] = pRow
                    pfieldsvalues = pRow
                    ShapeVal = [pfieldsvalues[4], pfieldsvalues[5]]
                    fillValues = [ShapeVal] + list(pfieldsvalues)
                    print(fillValues) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
                    newPoint.insertRow(fillValues)
                    print("point " + str(station) + " added to All_points")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

View solution in original post

DarrenWiens2 · ‎02-10-2017

Can you quickly post a screenshot of the attribute table showing the 17 points?

Nicole_Ueberschär · ‎02-11-2017

Here we go... (I hope it works with just copy-paste...)

DanPatterson_Retired · ‎02-10-2017

just in case looping through 18 objects would be the sequence 0 through 17 inclusive... for a total of 18

JoshuaBixby · ‎02-10-2017

It helps to err on the side of giving more code than less. You have provided what you think is the functional and broken part of the code, but it could be the issue is upstream on how some of the variables are created/defined.

Also, just a comment on using an insert cursor, create it outside of the loop where you insert records.

DanPatterson_Retired · ‎02-10-2017

I think this thread is related to this thread, which isn't closed

Nicole_Ueberschär · ‎02-11-2017

Yes, correct. Didn't want to mix up the issues. I separated now both parts - creating first the tables and then the points but now I am stuck here...

Nicole_Ueberschär · ‎02-11-2017

Didn't want to "overload" the code but here is the whole snipplet:

import arcpy
import os
#Workspace settings
arcpy.env.workspace=r"C:\data\2017\LakeKivu\Kivu_python.gdb"
arcpy.env.overwriteOutput= True

def unique_values(table, field):
    with arcpy.da.SearchCursor(table, [field]) as cursor:
        return sorted({row[0] for row in cursor})


#input file
table = r"C:\data\2017\LakeKivu\Kivu_python.gdb\Test1"

#Output files
DirOutProfiles=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Profiles"
DirOutCampaign=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Campaigns"
DirOutTables=r"C:\data\2017\LakeKivu\Kivu_python.gdb"
points=r"C:\data\2017\LakeKivu\Kivu_python.gdb\Campaigns\All_points"

profile_nrs=unique_values(table, "Station")
print(profile_nrs)

dsc = arcpy.Describe(table)
fields = dsc.fields
# List all field names except the OID field
fieldnames = [field.name for field in fields if field.name != dsc.OIDFieldName]

pfields=arcpy.ListFields(points)
fillnames=[pfield.name for pfield in pfields if pfield.name != dsc.OIDFieldName]  #names for point feature class

# get tables
for profile in profile_nrs:
    expression="Station="+str(profile)
    print(expression)
    with arcpy.da.SearchCursor(table,fieldnames,expression) as sCursor:
        for sRow in sCursor:
            table_name=str(sRow[0])+"_"+str(sRow[1])
        del sRow


# Create point feature
    #Identify first OBJECTID used in new table
    oid_nrs=unique_values(table_name, "OBJECTID")
    FirstObject=oid_nrs[0]
    print(FirstObject)
    #Define expression to limit the extracted rows to the first row
    pexpression="OBJECTID="+str(oid_nrs[0])

    #Define field names used for point table
    pfieldnames=['Cruise','Station', 'Type', 'Date', 'Longitude', 'Latitude','UPI']
    print(pfieldnames)

    #get values from table table_name
    pfieldsvalues=[]

    for pname in pfieldnames:
        with arcpy.da.SearchCursor(table_name, pname, pexpression) as pCursor:
         for pRow in pCursor:
            fillvalue=pRow[0]
            pfieldsvalues.append(fillvalue)
    ShapeVal=[pfieldsvalues[4],pfieldsvalues[5]]
    print(fillnames)
    fillValues=[ShapeVal, pfieldsvalues[0],pfieldsvalues[1],pfieldsvalues[2],pfieldsvalues[3],pfieldsvalues[4],pfieldsvalues[5],pfieldsvalues[6],]
    print(fillValues)
    newPoint=arcpy.da.InsertCursor(points, fillnames)
    newPoint.insertRow(fillValues)
    station_nr=pfieldsvalues[1]
    print("point "+str(station_nr) +" added to All_points")
    del pRow
    del pCursor
del sCursor‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I'm pretty sure that there is a better/simpler way of creating my points but I didn't find it yet.

Maybe I miss somewhere a point in these Cursor methodologies. I couldn't figure out how to take only a certain number of attributes from the input table for my point feature PLUS adding somehow the shape values that's why it might look a bit irritating...

In short what I do here:

- profile_nrs takes all unique numbers in the attribute station (1-18) from input table Table1 (rows 21)

- I created in a different script tables based on these unique numbers. This way I got 18 tables with the according names (concatenating attributes 1 and 2, resulting into something like bla_1, bla_2,...) (rows 33-39)

- take only the first row of each table for creating the point (->pexpression, row 48+58)

- write in pfieldvalues the values from the input table (in this case one of the new 18 tables).(rows 54-61)

- fillnames includes the SHAPE attribute

- fillvalues takes the created SHAPE attribute value (ShapeVal) plus the values written in pfieldvalues. (row 64)

- then insert a row into points with fillvalues. (row 66/67).

Let's hope this makes it somehow clearer...

DanPatterson_Retired · ‎02-11-2017

I am still having issues with the indentation, beginning a line 39 to the end... Is this a copy paste issue? it should be all dedented one level. There is still a whole load of cursor stuff going on

JoshuaBixby · ‎02-11-2017

After taking some time to read over the whole code, I think I get the gist of what you are attempting to do, but I don't get the implementation. For example, I am really confused by lines 53 through 58. You appear to be creating multiple search cursors, each defined with one field and one row, just to fill a list of values that represent a subset of fields of that one row. Why not bypass looping over pfieldnames and just pass pfieldnames to the search cursor (replace lines 51-61):

with arcpy.da.SearchCursor(table_name, pfieldnames, pexpression) as pCursor:
    pfieldsvalues = next(pCursor)
ShapeVal = [pfieldsvalues[4], pfieldsvalues[5]]
print(fillnames)
fillValues = [ShapeVal] + list(pfieldsvalues)
print(fileValues)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Also, the creation of the insert cursor (line 66) can be moved outside of the for loop that starts at line 33.