topic Re: Field calculation issue when using memory workspace in Python Snippets Questions

Field calculation issue when using memory workspace

TimothyMichael — Sun, 12 Dec 2021 08:03:02 GMT

Hi all,

I'm working on a script that uses the 'memory' workspace as layers are being processed, and I've hit an issue when I try to execute a field calculation between joined layers. I have included a script below that demonstrates the differences between workflows - the calculation will work fine if I create the feature layers from layers on disk, but will fail if the feature layers are created from layers in the memory workspace - testtype "fromdisk" works but testtype "frommemory" does not.

import arcpy

inLine = r"D:\TEMP\Output_20200520080600.gdb\LINE00240"
inPoint = r"D:\TEMP\Output_20200520080600.gdb\StructureJoin"

testtype = "fromdisk"

if testtype == "fromdisk":
    # Features are stored on disk
    # Feature layers are created from layers on disk
    
    print("Running disk workflow")

    # Make feature layers
    arcpy.MakeFeatureLayer_management(inLine, "LineFeature")
    arcpy.MakeFeatureLayer_management(inPoint, "StructureFeature")

    # Add join
    arcpy.AddJoin_management("LineFeature", "OBJECTID", "StructureFeature", "ORIG_FID", "KEEP_COMMON")

    # Calculate field
    in_field = "Field_Name"
    expression = '!StructureJoin.STRUCTURE_GUID!'
    arcpy.CalculateField_management("LineFeature", in_field, expression, "PYTHON3")

elif testtype == "frommemory":
    # Features are stored on disk
    # Make copy of data in memory
    # Feature layers are created from layers in memory

    print("Running memory workflow")
    
    # Copy features to memory
    arcpy.CopyFeatures_management(inLine, r"memory\Lines")
    arcpy.CopyFeatures_management(inPoint, r"memory\Structures")

    # Make feature layers
    arcpy.MakeFeatureLayer_management(r"memory\Lines", "LineFeature")
    arcpy.MakeFeatureLayer_management(r"memory\Structures", "StructureFeature")

    # Add join
    arcpy.AddJoin_management("LineFeature", "OBJECTID", "StructureFeature", "ORIG_FID", "KEEP_COMMON")

    # Calculate field
    in_field = "Field_Name"
    expression = '!Structures.STRUCTURE_GUID!'
    arcpy.CalculateField_management("LineFeature", in_field, expression, "PYTHON3")
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The resulting error is:

Running memory workflow
Traceback (most recent call last):
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\management.py", line 4527, in CalculateField
retval = convertArcObjectToPythonObject(gp.CalculateField_management(*gp_fixargs((in_table, field, expression, expression_type, code_block, field_type), True)))
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\geoprocessing\_base.py", line 511, in <lambda>
return lambda *args: val(*gp_fixargs(args, True))
arcgisscripting.ExecuteError: ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds.
Failed to execute (CalculateField).

Do I need to do something different when using the memory workspace?

Re: Field calculation issue when using memory workspace

JoeBorgione — Wed, 20 May 2020 13:56:12 GMT

I am a huge fan of Richard Fairhurst‌'s approach described here: /blogs/richard_fairhurst/2014/11/08/turbo-charging-data-manipulation-with-python-cursors-and-dictionaries and use it often. You may want to give it a try.

Re: Field calculation issue when using memory workspace

DavidPike — Wed, 20 May 2020 14:56:32 GMT

You could try utilising the arpy.env.scratchWorkspace e.g. output_path = os.path.join(arcpy.env.scratchWorkspace, output_name), it's also a useful workaround for GP tools which contain functions requiring physical outputs/copies, the server handles them in a pseudo in_memory capacity (as far as the user is concerned).

Re: Field calculation issue when using memory workspace

TimothyMichael — Wed, 20 May 2020 15:27:01 GMT

In the original version of this script I was writing everything out to disk first, and that worked fine, and I confirmed that the field calculation works that way. I'm trying to do as much in memory as possible to eliminate the disk I/O.

Re: Field calculation issue when using memory workspace

TimothyMichael — Wed, 20 May 2020 15:28:45 GMT

Thank you for this link - I'm going to give the dictionary example a try and see if that works with the in-memory features. Thank you for the pointer!

Re: Field calculation issue when using memory workspace

JoeBorgione — Wed, 20 May 2020 15:36:03 GMT

It's really cool; no memory, no join, just get the data from the source to the target: I have used it in memory, but only because I was updating in memory tables from other in memory tables where the data eventually found it's way to a feature class on disk.

Re: Field calculation issue when using memory workspace

JoshuaBixby — Wed, 20 May 2020 18:05:36 GMT

In-memory workspaces (memory and in_memory) are not full-fledged workspaces, some geodatabase features/functionality work, some don't. In this case, I do think it is a defect because I was able to use the older in_memory workspace in a standalone script and it works fine, no errors with expected results.

Re: Field calculation issue when using memory workspace

JoeBorgione — Wed, 20 May 2020 18:59:41 GMT

Seems like there have been other threads discussing some short comings of the memory workspace; I've never used it, just in_memory...

Re: Field calculation issue when using memory workspace

TimothyMichael — Thu, 21 May 2020 15:42:39 GMT

Using an example from the blog post that Joe shared I was able to get the calculation to work with the UpdateCursor in the memory workspace, so it looks like I'm good to proceed. I didn't test using the older in_memory workspace to see if my join + calculation method would work, but I'd rather get it to work with the newer workspace type.

Re: Field calculation issue when using memory workspace

JoeBorgione — Thu, 21 May 2020 15:50:49 GMT

I'm glad it worked for you; I can't tell you how many times I've used that approach!