topic Re: Moving values within FC works, but not for a List of FCs in Python Questions

Moving values within FC works, but not for a List of FCs

GabrielBacca-Cortes — Wed, 21 Jan 2015 19:37:56 GMT

Hi Everyone -

I have a code that moves values around within a FC [which is the result of an intersection] from one field to another so that I will have them all previously intersected values aligned in one single row. The code works well when I apply it to one FC at a time, but when I extend it to a List of FCs, it does not work. I've tried fixing it by playing with the indentation of the UpdateCursor lines, but it didn't work. Any idea what might be happening/missing ?

On another token, it seems to me that what I am trying to do, it could be done a lot simpler, any ideas on how to optimize it ?

Thanks much !

Gabriel

import arcpy, os

from arcpy import env

# Set environment settings

env.workspace = r"C:\GIS\Track1\1_Orig\int2"

# Set local variables

copyFields = ("PorousMed","K_1","Pity_1","Leng_m")

insertFd1 = ['PM1','K1','P1','leng_m1']

insertFd2 = ['PM2','K2','P2','leng_m2']

insertFd3 = ['PM3','K3','P3','leng_m3']

insertFd4 = ['PM4','K4','P4','leng_m4']

insertFd5 = ['PM5','K5','P5','leng_m5']

valueList = []

try:

fcList = arcpy.ListFeatureClasses("","POLYLINE")

for fc in fcList:

print fc

rows = arcpy.da.SearchCursor(fc, copyFields)

for row in rows:

valueList.append(row[0])

valueList.append(row[1])

valueList.append(row[2])

valueList.append(row[3])

cursor = arcpy.UpdateCursor(fc, insertFd1)

for row in cursor:

row.setValue("PM1",valueList[0])

row.setValue("K1",valueList[1])

row.setValue("poros1",valueList[2])

row.setValue("leng_m1",valueList[3])

cursor.updateRow(row)

del cursor, row

cursor = arcpy.UpdateCursor(fc, insertFd2)

for row in cursor:

row.setValue("PM2",valueList[4])

row.setValue("K2",valueList[5])

row.setValue("poros2",valueList[6])

row.setValue("leng_m2",valueList[7])

cursor.updateRow(row)

del cursor, row

cursor = arcpy.UpdateCursor(fc, insertFd3)

for row in cursor:

row.setValue("PM3",valueList[8])

row.setValue("K3",valueList[9])

row.setValue("poros3",valueList[10])

row.setValue("leng_m3",valueList[11])

cursor.updateRow(row)

del cursor, row

cursor = arcpy.UpdateCursor(fc, insertFd4)

for row in cursor:

row.setValue("PM4",valueList[12])

row.setValue("K4",valueList[13])

row.setValue("poros4",valueList[14])

row.setValue("leng_m4",valueList[15])

cursor.updateRow(row)

del cursor, row

cursor = arcpy.UpdateCursor(fc, insertFd5)

for row in cursor:

row.setValue("PM5",valueList[16])

row.setValue("K5",valueList[17])

row.setValue("poros5",valueList[18])

row.setValue("leng_m5",valueList[19])

cursor.updateRow(row)

del cursor, row

except:

print arcpy.GetMessages(0)

print arcpy.GetMessages(1)

print arcpy.GetMessages(2)

Re: Moving values within FC works, but not for a List of FCs

JoshuaBixby — Wed, 21 Jan 2015 20:45:47 GMT

At first glance, you seem to be mixing older/original cursors and the newer data access (arcpy.da) cursors. Your syntax for the update cursors seems wrong. The arcpy.da.UpdateCursor syntax is (in_table, field_names, {where_clause}, {spatial_reference}, {explode_to_points}, {sql_clause}) whereas the arcpy.UpdateCursor syntax is (dataset, {where_clause}, {spatial_reference}, {fields}, {sort_fields}). You appear to be passing your field list as a where clause to your update cursors.

Overall, I recommend switching to using all arcpy.da cursors.

Re: Moving values within FC works, but not for a List of FCs

GabrielBacca-Cortes — Thu, 22 Jan 2015 03:50:14 GMT

Thanks Joshua for your response. I tried what you suggested but didn't solve my main problem which is the iteration of this procedure through the other FCs. Actually, when I switched everything back to the newer data access version, it didn't work at all not even for the first FC. I know it is not ideal having cursors with DA and without it, but it gives me the result I want, only that it can be replicated with the list of FCs.

thanks again,

Gabriel

Re: Moving values within FC works, but not for a List of FCs

SuryaKant — Thu, 22 Jan 2015 06:57:56 GMT

Hi Gabriel,
I tried to understand your script but it it very confusing(please don't mind). Could you explain your work process(details are important). What are your inputs and what would be your expected result.
This will help me to write a script for you.
Surya

Re: Moving values within FC works, but not for a List of FCs

XanderBakker — Thu, 22 Jan 2015 12:35:36 GMT

Yeah, some explanation on the general idea would be helpful. From what I see in your code it doesn’t make much sense:

Your create a list of polyline featureclasses in you workspace
You loop through the featureclasses
- for each feature in the current featureclass you append the values of the input fields to a list
- next you create an update cursor on the same featureclass and start to set the values of the output fields. When doing this you assign the first 4 values of the list (not the additional values appended to the list).

Please note that for the second feature in the featureclass (da.searchcursor) you add the values to the list, but the values from the 1st feature will still be assigned. I don’t think that is what you want to achieve, right?

Since this is all happing in the same featureclass, you could use a single da.UpdateCursor to do this (assuming that you want to assign the value from each feature input field to the output fields of the same feature. That does make sense, although I don’t see why you would want to duplicate those values 5 times in your featureclass...

Re: Moving values within FC works, but not for a List of FCs

GabrielBacca-Cortes — Thu, 22 Jan 2015 15:39:33 GMT

Hi Xander and Surya: Thanks for your response.

The two bullets in which Xander summarized part of my code are correct. I am sorry I should have provided a better explanation on my first post. The list of values I am creating is a 5rx4c table that I can index out as follows, so that I can move values to different fields within the same FC as:

[ 0 1 2 3 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

4 5 6 7 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

16 17 18 19] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

The values in the original file, in vertical order in the input fields, are the result of intersecting a polyline against polygons, but the main goal of this code is to have a one single part polyline that includes all the information from the intersection. I need to do this for thousands of intersected polylines {FCs} and then append them all out, but that's not in the code I posted. In order to get a one single part polyline I will dissolve the FC for the output fields.

So, yes, the valueList will have to be re-created for each FC, to re-distribute the proper values within it as indicated above.

The code, as it is, does the job for the first FC but it stalls and does not move on to the next FC.

Let me know if you have further questions. Thanks a bunch for comments !

Gabriel

Re: Moving values within FC works, but not for a List of FCs

JoshuaBixby — Thu, 22 Jan 2015 16:36:16 GMT

By stall out, what exactly do you mean? Does it stop running and hang? Does it complete but not update what you expect? Does it stop running and generate an error?

Re: Moving values within FC works, but not for a List of FCs

GabrielBacca-Cortes — Thu, 22 Jan 2015 16:43:44 GMT

Joshua - by stall out I mean that it does not move to the next FC and locks the modified file, that is the first FC, without generating an error,

regards,

Gabriel

Re: Moving values within FC works, but not for a List of FCs

JoshuaBixby — Thu, 22 Jan 2015 17:06:02 GMT

Regarding your use of the older/original UpdateCursor, you are using incorrect syntax. You are getting lucky, if one wants to call it that, because there is a bug in the UpdateCursor where passing a Python list instead of a text-based where_clause causes the where_clause to be ignored. In effect, using cursor = arcpy.UpdateCursor(fc) generates the same cursor for you.

The fact that switching everything to the Data Access cursors meant the code didn't work at all, not even the first FC, is telling you something is wrong with the code even for the first FC. What exactly happened when you use arcpy.da UpdateCursors? Does it run and nothing happens? An error code is generated?

Re: Moving values within FC works, but not for a List of FCs

JoshuaBixby — Thu, 22 Jan 2015 17:17:01 GMT

Since you are using a search cursor and update cursor against the FC at the same time, and the search cursor creates the first set of locks, it may be causing an issue since the update cursor will want to generate more locks. Not sure, just thinking aloud.

One thing to try, put a del rows statement at the bottom of your try statement (indented to the same level as your search cursor statement). Deleting the search cursor explicitly will more cleanly remove the locks before you move onto the next FC. In theory, reassigning the rows variable to a new search cursor object should release the locks on the earlier search cursor, but that may not be happening cleanly.

Re: Moving values within FC works, but not for a List of FCs

JoshuaBixby — Thu, 22 Jan 2015 17:23:57 GMT

Bigger picture, you don't need to be using both search cursors and update cursors since both cursors are open on the same FC. You can iterate over an update cursor without updating any values, in effect using it like a search cursor. Usually you wouldn't want to do that because it creates more locks and may have a bit more overhead in terms of performance. In this case, since you are going to open update cursors on the FC already, just work with update cursors.

Furthermore, you should be able to restructure the code to do everything with a single update cursor. Instead of creating a new update cursor each time, since your insertFD lists aren't doing anything anyways, you can just reset the cursor and iterate over it again. Resetting a cursor and reusing it is much more efficient and clean in terms of locks than recreating new cursors every time.

Re: Moving values within FC works, but not for a List of FCs

XanderBakker — Sat, 11 Dec 2021 06:50:30 GMT

Sorry Gabriel, it is still unclear what you mean exactly, but if I have to rewrite what your original code was trying to do, then it would be something like this.

import arcpy, os
from arcpy import env

# Set environment settings
env.workspace = r"C:\GIS\Track1\1_Orig\int2"

# Set local variables
dct_flds = {'PorousMed': ['PM1','PM2','PM3','PM4','PM5'],
            'K_1': ['K1','K2','K3','K4','K5'],
            'Pity_1': ['P1','P2','P3','P4','P5'],
            'Leng_m': ['leng_m1','leng_m2','leng_m3','leng_m4','leng_m5']}

# create list of fields from dictionary (contains all fields: input and output)
flds = []
for k, lst in dct_flds.items():
    flds.append(k)
    for fld in lst:
        flds.append(fld)

try:
    fcList = arcpy.ListFeatureClasses("","POLYLINE")
    for fc in fcList:
        print fc
        with arcpy.da.UpdateCursor(fc, flds) as curs:
            for row in rows:
                for fld_in, lst_out in dct_flds.items():
                    val_in = row[flds.index(fld_in)]
                    for fld_out in lst_out:
                        row[flds.index(fld_out)] = val_in
                curs.updateRow(row)

except:
    print arcpy.GetMessages(0)
    print arcpy.GetMessages(1)
    print arcpy.GetMessages(2)

Re: Moving values within FC works, but not for a List of FCs

GabrielBacca-Cortes — Thu, 22 Jan 2015 18:11:03 GMT

Thanks Xander - I ran the script you sent, but didn't run. The [flds] list looks very different from what I need. So I am hoping that by adding the fields on top here, it is clearer how the code should deliver the results I want.

PM PM1 PM2 pm5

K_1 K1 K2 k5

Pit_1 P1 P2 p5

LenM Lm1 Lm2 lm5

[ 0 1 2 3 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

4 5 6 7 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

16 17 18 19] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

After dissolving I will be getting:

PM1 PM2 pm5

K1 K2 k5

P1 P2 p5

Lm1 Lm2 lm5

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Re: Moving values within FC works, but not for a List of FCs

XanderBakker — Thu, 22 Jan 2015 18:25:16 GMT

OK, I think I am starting to get the picture when there are 5 features. But with if the featureclass has more or less, what should happen then?

Re: Moving values within FC works, but not for a List of FCs

XanderBakker — Sat, 11 Dec 2021 06:50:33 GMT

The best thing would be to share some sample data. That would make it so much easier. My next best guess is this (not testes and should best be run on a copy of the data):

import arcpy, os
from arcpy import env

# Set environment settings
env.workspace = r"C:\GIS\Track1\1_Orig\int2"

# Set local variables
dct_flds = {'PorousMed': ['PM1','PM2','PM3','PM4','PM5'],
            'K_1': ['K1','K2','K3','K4','K5'],
            'Pity_1': ['P1','P2','P3','P4','P5'],
            'Leng_m': ['leng_m1','leng_m2','leng_m3','leng_m4','leng_m5']}

# create list of fields from dictionary (contains all fields: input and output)
flds = []
for k, lst in dct_flds.items():
    flds.append(k)
    for fld in lst:
        flds.append(fld)

flds_in = dct_flds.keys()

try:
    fcList = arcpy.ListFeatureClasses("","POLYLINE")
    for fc in fcList:
        print fc
        # create the list from the input fields
        valueList = []
        with arcpy.da.SearchCursor(fc, flds_in) as curs:
            for row in curs:
                valueList.append(list(row))

        with arcpy.da.UpdateCursor(fc, flds) as curs:
            for row in rows:
                for fld_in, lst_out in dct_flds.items():
                    # val_in = row[flds.index(fld_in)]
                    for fld_out in lst_out:
                        num = int(fld_out[-1:])
                        lst_vals = list(valueList[num])
                        row[flds.index(fld_out)] = lst_vals[flds_in.index(fld_in)]
                curs.updateRow(row)

except:
    print arcpy.GetMessages(0)
    print arcpy.GetMessages(1)
    print arcpy.GetMessages(2)

Re: Moving values within FC works, but not for a List of FCs

GabrielBacca-Cortes — Thu, 22 Jan 2015 19:37:41 GMT

Thanks Xander - not yet... I am attaching a sample set of 5 FCs together with a copy of my code and a copy of your code. Please bear in mind that for simplicity I had changed the names of the fields in the version of the code I've posted here in the forum.

The first FC "xy_1.shp" includes the results of running my code. Tthat's the way it should look for the rest of FCs.

I just noticed that you are in Medellin. I am from Colombia, my family lives in Cali, just came back last week actually.

Thanks again for your help on this !

Regards,

Gabriel

Re: Moving values within FC works, but not for a List of FCs

GabrielBacca-Cortes — Thu, 22 Jan 2015 19:41:45 GMT

if the feature Class has more, those values do not get transfered, if it has less, their value are empty for strings/text or zero.

I'll likely in the future increase the number from 5 to 10 just to be safe but it will not be likely to have more than 8-10. I just need something that works first...

thanks !

Re: Moving values within FC works, but not for a List of FCs

SusanJones — Thu, 22 Jan 2015 21:47:54 GMT

I had exactly this last week with using the arcpy.da.InsertCursor and arcpy.da.SearchCursor.

Seems that can only use one cursor at a time.

In the end I pulled all my search records into an array and used them as part of my insert/update records.

Took a bit of getting my head around, but got there in the end.

Here is my logic:

#todo: process Layer

print "\nProcess CAFS Layer"

#todo: search Data

recs = arcpy.da.SearchCursor(in_table = "EMU_VIEW", field_names = ["VehicleId", "Timestamp", "Longitude", "Latitude"])

rows = []

for rec in recs:

pt = [rec[2], rec[3]]

row = [pt, rec[1], rec[0], "CAFS", rec[1], rec[2], rec[3]]

rows.append(row)

del recs

#todo: insert Data

print "insert Data"

fldList = ["SHAPE@XY", "DateTime", "VehicleID", "FeedType", "Timestamp", "Longitude", "Latitude"]

irecs = arcpy.da.InsertCursor(in_table = fc, field_names = fldList)

for row in rows:

irecs.insertRow(row)

del irecs

Susan

Re: Moving values within FC works, but not for a List of FCs

JoshuaBixby — Thu, 22 Jan 2015 21:59:07 GMT

I wouldn't go as far as saying you can only use one cursor at a time. There are several factors that affect how and whether multiple cursors will work together on a data set. The types of locks that the cursors require are one factor. The order of which cursors are opened first can be a factor. The backend data storage format is another factor. When it comes to using multiple cursors to access spatial data, having the data in an enterprise DBMS makes a big difference.

Re: Moving values within FC works, but not for a List of FCs

SusanJones — Thu, 22 Jan 2015 22:06:32 GMT

Totally agree - storage format is part of it. For tis particular task I have been using File Geodatbase