efficient code for field calculation

[RavenSchmidt]

Hi, I would like to calculate a single layer in one field from the value of another field. The problem lies within the size of the dataset. My Computer is very fast, but the calculation seems to slow down considerably over time until it barely moves anymore. I stopped it after three days.
Here are two versions of the codeblock I have tried so far.
Does anybody have suggestions how to calculate the field more efficiently?
Curiouse side note: I used this code several times already and sometimes it just seems to be significantly faster as on other times with no major changes in background load on the system...
Also this seems to apply both to running the code in PyCharm and ArcGIS Pro Field Calculator

A

expression = "hierarchy_creator_feet(!highway!)"
codeblock = """def hierarchy_creator_feet(p_highway):
    if p_highway in ('footway', 'pedestrian', 'residential', 'living_street', 'crossing', 'track', 'path'):
        return 3
    elif p_highway in ('tertiary', 'tertiary_link', 'unclassified', 'steps', 'service'):
        return 23
    elif p_highway in ('secondary', 'secondary_link', 'primary', 'road'):
        return 33
    elif '[' in p_highway: return 32
    else:
        return 0"""

B

highway_hierarchy = {
    'footway': 3, 
    'pedestrian': 3, 
    'residential': 3, 
    'living_street': 3, 
    'crossing': 3, 
    'track': 3, 
    'path': 3,
    'tertiary': 23, 
    'tertiary_link': 23, 
    'unclassified': 23, 
    'steps': 23, 
    'service': 23,
    'secondary': 33, 
    'secondary_link': 33, 
    'primary': 33, 
    'road': 33
}

expression = "hierarchy_creator_feet(!highway!)"
codeblock = """def hierarchy_creator_feet(p_highway):
    return highway_hierarchy.get(p_highway, 0)"""

[Eugene_Adkins]

If your data source comes in through Portal and you’re using Pro, I have found that using calculator in webmap viewer (classic) works much faster with larger datasets. Just make sure everything is setup the way you want it because there is no undo with webmap viewer.

If you don’t use Portal then obviously this workaround probably won’t apply.

[RavenSchmidt]

Thank you, sadly this is not an option for me

[RPGIS]

The code above looks simple enough that it is efficient as is. If you are looking to simplify your code further you can use this example below:

Expression:

hierarchy_creator_feet(!highway!)

Code Block:

def hierarchy_creator_feet(p_highway):
    hierarchy = 0
    if p_highway in ('footway', 'pedestrian', 'residential', 'living_street', 'crossing', 'track', 'path'): hierarchy = 3
    elif p_highway in ('tertiary', 'tertiary_link', 'unclassified', 'steps', 'service'): hierarchy = 23
    elif p_highway in ('secondary', 'secondary_link', 'primary', 'road'): hierarchy = 23
    elif '[' in p_highway: hierarchy = 33
    return hierarchy

or

def hierarchy_creator_feet(p_highway):
    hierarchies = {
        3 : ('footway', 'pedestrian', 'residential', 'living_street', 'crossing', 'track', 'path'),
        23 : ('tertiary', 'tertiary_link', 'unclassified', 'steps', 'service'),
        33 : ('secondary', 'secondary_link', 'primary', 'road')
        32 : '['
        }
    hierarchy = 0
    for a , b in hierarchies.items():
        if p_highway in b or b in p_highway: hierarchy = a
    return hierarchy

If you are able to reduce your code to the simplest level for both the result and ease of troubleshooting then it is already efficient enough. There are ways to write simpler code, but it mostly boils down to preference at that point.

[RavenSchmidt]

Thank you

[JoshuaBixby]

The issue likely has to do with the data store or workspace and not your code.  Are you updating a feature service, enterprise geodatabase feature class, shape file, etc...?

[RavenSchmidt]

God Morgon!
I am updating a fileGDB Feature Class.
Should I load it in Memory, do the calculation and save it again afterwards?

EDIT: I just tried that, but I can't overwrite the FC with copy features in the end because it is connected to a Network Dataset and therefore locked

[RPGIS]

In memory simply creates a temporary feature. The feature itself is unrelated to the actual feature that the data is created from hence in memory. The processing would be roughly the same, though it largely depends on the type of cpu that you are using. If speed is something you are looking for then you could check out threading and multiprocessing.

[RavenSchmidt]

Well I can copy the FC into memory, do all the calculations in memory and overwrite the original afterwards. When I tried it the calculations were about 100x faster. The scope of the task is the complete road/way-network for Germany, so over 40.000.000 features, where I intend to change attributes that relate to the walkability of segments. In memory the whole thing completed in 1 to 2 hours. Froms my SSD, which is a Samsung 990 Pro, so nothing to complain about, the calculation process got progressively slow. In the beginning it calculated say 10% in an hour, after 3 days it was say 1% every 3 hours...

[JoshuaBixby]

Is the FGDB feature class hosted on a local drive on your machine or a network share?