topic Re: Best Way to Calculate Many Fields in Python Questions

Best Way to Calculate Many Fields

GeoffOlson — Thu, 19 Sep 2013 15:46:06 GMT

Short Explanation: I have 8 fields I need to calculate that uses 2 fields as variables as to designate which of the 8 fields to calculate.

Long Explanation: I have a streets feature class that I need to be able to update traffic capacity for. There are 4 intervals (2010, 2019, 2025, and 2045) and the capacity is calculated with directionality, so for each year interval there are 2 capacity fields - 8 fields total. Each street feature has a field for each year interval that specifies if it's one-way (1 or -1 depending on how the feature was drawn) or two-way. If it's a two-way street then both fields are calculated and if it's one-way then the direction field dictates which field to calculate for. Easy enough. To complicate that a little more I have a field that needs to specify which years street is "valid" for and that dictates which years to calculate capacity for.

So far I have my calculation function working; it's my python field calculator script. I also have a single year calculating properly. So the road name, function, area type, and number of lanes are what I'm using to calculate capacity. I'm using an updateCursor to retrieve data values and assign the calculated values.

Disregarding the function for now, here's what I have:

import arcpy, os, sys

#set map doc and the layer to be used
mxd = arcpy.mapping.MapDocument("Current")
mapLyr = arcpy.mapping.ListLayers(mxd, "CUR_Master_Network_2045")[0]
rows = arcpy.UpdateCursor(mapLyr)

for row in rows:
    row.F2010_AB_C = 0 #clear all capacities to 0
    row.F2010_BA_C = 0
    row.F2019_AB_C = 0
    row.F2019_BA_C = 0
    row.F2025_AB_C = 0
    row.F2025_BA_C = 0
    row.F2045_AB_C = 0
    row.F2045_BA_C = 0
    rows.updateRow(row)
    #find capacity variables
    rnam = row.getValue("ROAD_NAM")
    ffc =  row.getValue("LRTP_FFC")
    at = row.getValue("TYPE_AREA")
    lan = row.getValue("F2010_NUML")
    dic = {"ROAD_NAM": rnam, "LRTP_FFC": ffc, "TYPE_AREA": at}
    #begin capacity calculation
    if row.BUILD_STAT == 10 and row.F2010_DIR == 1:
            row.F2010_AB_C = capacity(**dic) * lan
            rows.updateRow(row)
    elif row.BUILD_STAT == 10 and row.F2010_DIR == -1:
            row.F2010_AB_C = capacity(**dic) * lan
            rows.updateRow(row)
    elif row.BUILD_STAT == 10 and row.F2010_DIR == 0:
            row.F2010_AB_C = capacity(**dic) * lan
            row.F2010_BA_C = capacity(**dic) * lan
            rows.updateRow(row)
    else:
        pass
del row, rows

This is working great, but it's only for a single set of parameters - BUILD_STAT 10 and the three possible directions. What's the best way to add code to calculate not only BUILD_STAT 10 but also 19, 25, and 45 and follow the directions? Do I just need to keep nesting if statements for each BUILD_STAT year?

import arcpy, os, sys

#set map doc and the layer to be used
mxd = arcpy.mapping.MapDocument("Current")
mapLyr = arcpy.mapping.ListLayers(mxd, "CUR_Master_Network_2045")[0]
rows = arcpy.UpdateCursor(mapLyr)

for row in rows:
    row.F2010_AB_C = 0 #clear all capacities to 0
    row.F2010_BA_C = 0
    row.F2019_AB_C = 0
    row.F2019_BA_C = 0
    row.F2025_AB_C = 0
    row.F2025_BA_C = 0
    row.F2045_AB_C = 0
    row.F2045_BA_C = 0
    rows.updateRow(row)
    #find capacity variables
    rnam = row.getValue("ROAD_NAM")
    ffc =  row.getValue("LRTP_FFC")
    at = row.getValue("TYPE_AREA")
    lan = row.getValue("F2010_NUML")
    dic = {"ROAD_NAM": rnam, "LRTP_FFC": ffc, "TYPE_AREA": at}
    #begin capacity calculation
    if row.BUILD_STAT == 10 and row.F2010_DIR == 1:
            row.F2010_AB_C = capacity(**dic) * lan
            rows.updateRow(row)
            if row.F2019_DIR == 1:
                row.F2019_AB_C = capacity(**dic) * lan
                rows.updateRow(row)
            elif row.F2019_DIR == -1:
                row.F2019_BA_C = capacity(**dic) * lan
                rows.updateRow(row)
            else:
                row.F2019_AB_C = capacity(**dic) * lan
                row.F2019_BA_C = capacity(**dic) * lan
    elif row.BUILD_STAT == 10 and row.F2010_DIR == -1:
            row.F2010_AB_C = capacity(**dic) * lan
            rows.updateRow(row)
            if row.F2019_DIR == 1:
                row.F2019_AB_C = capacity(**dic) * lan
                rows.updateRow(row)
            elif row.F2019_DIR == -1:
                row.F2019_BA_C = capacity(**dic) * lan
                rows.updateRow(row)
            else:
                row.F2019_AB_C = capacity(**dic) * lan
                row.F2019_BA_C = capacity(**dic) * lan
    elif row.BUILD_STAT == 10 and row.F2010_DIR == 0:
            row.F2010_AB_C = capacity(**dic) * lan
            row.F2010_BA_C = capacity(**dic) * lan
            rows.updateRow(row)
            if row.F2019_DIR == 1:
                row.F2019_AB_C = capacity(**dic) * lan
                rows.updateRow(row)
            elif row.F2019_DIR == -1:
                row.F2019_BA_C = capacity(**dic) * lan
                rows.updateRow(row)
            else:
                row.F2019_AB_C = capacity(**dic) * lan
                row.F2019_BA_C = capacity(**dic) * lan
    else:
        pass
del row, rows

The only problem is each if and elif will need every possible outcome as it goes through the conditions. Is there a better way?

Re: Best Way to Calculate Many Fields

StacyRendall1 — Sat, 11 Dec 2021 10:28:53 GMT

The first thing I notice about your code is that there is a lot of repetition... Your first two conditions do exactly the same thing, and the third condition repeats it. Else: pass is not required. updateRow won't hurt anything if you have made no changes. You can also use in to test multiple conditions. This should work replicate the behaviour of your first code block (sorry, no way to test it!):

    #begin capacity calculation
    if (row.BUILD_STAT == 10) and (row.F2010_DIR in [1, -1]):
        row.F2010_AB_C = capacity(**dic) * lan
    elif (row.BUILD_STAT == 10) and (row.F2010_DIR == 0):
        row.F2010_AB_C = capacity(**dic) * lan
        row.F2010_BA_C = capacity(**dic) * lan
    rows.updateRow(row)

del row, rows

Or was your repetition of row.F2010_AB_C = capacity(**dic) * lan a mistake?

For your more advanced one you should note that each row only has one build_stat value, either 10, 19, 25 or 45. Furthermore you can use setValue to set a value by name. This should do what you want for your second code block (sorry, no way to test it!):

import arcpy, os, sys

#set map doc and the layer to be used
mxd = arcpy.mapping.MapDocument("Current")
mapLyr = arcpy.mapping.ListLayers(mxd, "CUR_Master_Network_2045")[0]
rows = arcpy.UpdateCursor(mapLyr)

for row in rows:
    row.F2010_AB_C = 0 #clear all capacities to 0
    row.F2010_BA_C = 0
    row.F2019_AB_C = 0
    row.F2019_BA_C = 0
    row.F2025_AB_C = 0
    row.F2025_BA_C = 0
    row.F2045_AB_C = 0
    row.F2045_BA_C = 0
    #find capacity variables
    rnam = row.getValue("ROAD_NAM")
    ffc =  row.getValue("LRTP_FFC")
    at = row.getValue("TYPE_AREA")
    lan = row.getValue("F2010_NUML")
    dic = {"ROAD_NAM": rnam, "LRTP_FFC": ffc, "TYPE_AREA": at}
    #capacity calculation
    yr = row.BUILD_STAT # any row can only have one year state...
    if row.getValue('row.F20%s_DIR' % yr) in [1, -1]:
        row.setValue('F20%s_AB_C' % yr, capacity(**dic) * lan)
    elif row.getValue('F20%s_DIR' % yr) == 0:
        row.setValue('F20%s_AB_C' % yr, capacity(**dic) * lan)
        row.setValue('F20%s_BA_C' % yr, capacity(**dic) * lan)
    rows.updateRow(row)

del row, rows

Re: Best Way to Calculate Many Fields

GeoffOlson — Sat, 11 Dec 2021 10:28:56 GMT

Thank you for the reply, Stacy. Actually, the ..DIR fields or 1 and -1 have different results, 1 will calculate for A to B flow of traffic and -1 will calculate for B to A flow. I used else in place of setting the condition to 0 because it's the only other value in that field. Your idea using yr as a variable is excellent, I never would have thought of that. That would work for the majority of my network, but I have to yr variables that do not have a corresponding 'F20%s' value. I originally tried using setValue, but it wouldn't update my attribute values. When I had success with updateCursor I decided to work with that. I did get the whole thing working stretched out with IF statements, and it didn't take as long as I thought it would because it only needed a few value changes with the correct indentation. It's just really long, 868 lines. Since at first when I was just testing to get it working I forgot that each calculation year has a corresponding NUML year, I just had to add three more variables. I thought it would take longer to run on my network which is about 7,900 features, and it only takes about 10 seconds. It's definitely way better than using my capacity calculation in the field calculator tool and manually defining queries and working within the attribute table.

There are no input parameters, all the variables are hard coded. Here's my final script:

import arcpy, os, sys

#set map doc and the layer to be used
mxd = arcpy.mapping.MapDocument("Current")
mapLyr = arcpy.mapping.ListLayers(mxd, "CUR_Master_Network_2045")[0]

#Single Lane Capacity Generator
def capacity(ROAD_NAM, LRTP_FFC, TYPE_AREA):
    if ROAD_NAM == "Interstate 74 Bridge":
        return 19000
    elif ROAD_NAM == "Interstate 80 Bridge":
        return 18000
    elif ROAD_NAM == "Interstate 280 Bridge":
        return 19000
    elif ROAD_NAM == "Centennial Bridge":
        return 11000
    elif ROAD_NAM == "Arsenal Bridge":
        return 9500
    
    elif LRTP_FFC == 1:
        return 19000

    elif LRTP_FFC == 2:
        if TYPE_AREA == 1:
            return 8500
        elif TYPE_AREA == 2:
            return 8650
        elif TYPE_AREA == 3:
            return 9000
        elif TYPE_AREA == 4:
            return 9000
        elif TYPE_AREA == 5:
            return 8000
        else:
            pass

    elif LRTP_FFC == 3:
        if TYPE_AREA == 1:
            return 8200
        elif TYPE_AREA == 2:
            return 8300
        elif TYPE_AREA == 3:
            return 8500
        elif TYPE_AREA == 4:
            return 8500
        elif TYPE_AREA == 5:
            return 7500
        else:
            pass

    elif LRTP_FFC == 4:
        if TYPE_AREA == 1:
            return 6300
        elif TYPE_AREA == 2:
            return 7500
        elif TYPE_AREA == 3:
            return 7500
        elif TYPE_AREA == 4:
            return 7500
        elif TYPE_AREA == 5:
            return 7500
        else:
            pass

    elif LRTP_FFC == 5:
        if TYPE_AREA == 1:
            return 5300
        elif TYPE_AREA == 2:
            return 6500
        elif TYPE_AREA == 3:
            return 6500
        elif TYPE_AREA == 4:
            return 6500
        elif TYPE_AREA == 5:
            return 5700
        else:
            pass

    elif LRTP_FFC == 6:
        return 15200

    elif LRTP_FFC == 7:
        if TYPE_AREA ==1:
            return 4800
        elif TYPE_AREA == 2:
            return 6000
        elif TYPE_AREA == 3:
            return 6000
        elif TYPE_AREA == 4:
            return 6000
        elif TYPE_AREA == 5:
            return 5200
        else:
            pass

    elif LRTP_FFC == 8:
        return 99999
        
    else:
        return 0

Re: Best Way to Calculate Many Fields

GeoffOlson — Sat, 11 Dec 2021 10:28:59 GMT

Here's one entire block for BUILD_STAT 10 and DIR 1

rows = arcpy.UpdateCursor(mapLyr)

for row in rows:
    row.F2010_AB_C = 0 #clear all capacities to 0
    row.F2010_BA_C = 0
    row.F2019_AB_C = 0
    row.F2019_BA_C = 0
    row.F2025_AB_C = 0
    row.F2025_BA_C = 0
    row.F2045_AB_C = 0
    row.F2045_BA_C = 0
    rows.updateRow(row)
    #find capacity variables
    rnam = row.getValue("ROAD_NAM")
    ffc =  row.getValue("LRTP_FFC")
    at = row.getValue("TYPE_AREA")
    lan10 = row.getValue("F2010_NUML")
    lan19 = row.getValue("F2019_NUML")
    lan25 = row.getValue("F2025_NUML")
    lan45 = row.getValue("F2045_NUML")
    dic = {"ROAD_NAM": rnam, "LRTP_FFC": ffc, "TYPE_AREA": at}
    #begin capacity calculation
    if row.BUILD_STAT == 10 and row.F2010_DIR == 1:
            row.F2010_AB_C = capacity(**dic) * lan10
            rows.updateRow(row)
            if row.F2019_DIR == 1:
                row.F2019_AB_C = capacity(**dic) * lan19
                rows.updateRow(row)
                if row.F2025_DIR == 1:
                    row.F2025_AB_C = capacity(**dic) * lan25
                    rows.updateRow(row)
                    if row.F2045_DIR == 1:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    elif row.F2045_DIR == -1:
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    else:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                elif row.F2025_DIR == -1:
                    row.F2025_BA_C = capacity(**dic) * lan25
                    rows.updateRow(row)
                    if row.F2045_DIR == 1:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    elif row.F2045_DIR == -1:
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    else:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                else:
                    row.F2025_AB_C = capacity(**dic) * lan25
                    row.F2025_BA_C = capacity(**dic) * lan25
                    rows.updateRow(row)
                    if row.F2045_DIR == 1:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    elif row.F2045_DIR == -1:
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    else:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
            elif row.F2019_DIR == -1:
                row.F2019_BA_C = capacity(**dic) * lan19
                rows.updateRow(row)
                if row.F2025_DIR == 1:
                    row.F2025_AB_C = capacity(**dic) * lan25
                    rows.updateRow(row)
                    if row.F2045_DIR == 1:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    elif row.F2045_DIR == -1:
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    else:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                elif row.F2025_DIR == -1:
                    row.F2025_BA_C = capacity(**dic) * lan25
                    rows.updateRow(row)
                    if row.F2045_DIR == 1:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    elif row.F2045_DIR == -1:
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    else:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                else:
                    row.F2025_AB_C = capacity(**dic) * lan25
                    row.F2025_BA_C = capacity(**dic) * lan25
                    if row.F2045_DIR == 1:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    elif row.F2045_DIR == -1:
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    else:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
            else:
                row.F2019_AB_C = capacity(**dic) * lan19
                row.F2019_BA_C = capacity(**dic) * lan19
                rows.updateRow(row)
                if row.F2025_DIR == 1:
                    row.F2025_AB_C = capacity(**dic) * lan25
                    rows.updateRow(row)
                    if row.F2045_DIR == 1:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    elif row.F2045_DIR == -1:
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    else:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                elif row.F2025_DIR == -1:
                    row.F2025_BA_C = capacity(**dic) * lan25
                    rows.updateRow(row)
                    if row.F2045_DIR == 1:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    elif row.F2045_DIR == -1:
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    else:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                else:
                    row.F2025_AB_C = capacity(**dic) * lan25
                    row.F2025_BA_C = capacity(**dic) * lan25
                    if row.F2045_DIR == 1:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    elif row.F2045_DIR == -1:
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)
                    else:
                        row.F2045_AB_C = capacity(**dic) * lan45
                        row.F2045_BA_C = capacity(**dic) * lan45
                        rows.updateRow(row)

I realize how long it is, but it does work and that's what matters most to me. But since I'm still pretty new to scripting I'm very happy to learn other ways of coding. I have a little brushing up to do on matrices, but is it possible to replace my entire capacity function with a matrix or table and under the specific conditions a cell value is returned?

Re: Best Way to Calculate Many Fields

StacyRendall1 — Mon, 23 Sep 2013 20:02:37 GMT

I realize how long it is, but it does work and that's what matters most to me. But since I'm still pretty new to scripting I'm very happy to learn other ways of coding. I have a little brushing up to do on matrices, but is it possible to replace my entire capacity function with a matrix or table and under the specific conditions a cell value is returned?

It's not exactly the length of the code that is the problem, but the fact you do the exact same thing 400 times, that is:

 row.F[YEAR]_[DIR]_C = capacity(**dic) * lan[YEAR]

It would not be particularly hard to write code to do what you want in about 20 lines... There are three main problems with your code at the moment that will affect you:

it is very easy to make mistakes

extremely hard to locate mistakes

time consuming and difficult to make changes.

To write code you need to figure out exactly what you are trying to do each step. I mean get a piece of paper and think the actual calculation steps through.

Re: Best Way to Calculate Many Fields

GeoffOlson — Mon, 23 Sep 2013 20:29:55 GMT

You're right. I just don't know of another way to loop through each combination to perform all the calculations, and that was the best I could do. It is long, and it was a lot to copy and paste and make changes to, but like I said, I don't know how else to direct the script through each possibility.