Hi all,
I'm working on a script that uses the 'memory' workspace as layers are being processed, and I've hit an issue when I try to execute a field calculation between joined layers. I have included a script below that demonstrates the differences between workflows - the calculation will work fine if I create the feature layers from layers on disk, but will fail if the feature layers are created from layers in the memory workspace - testtype "fromdisk" works but testtype "frommemory" does not.
import arcpy
inLine = r"D:\TEMP\Output_20200520080600.gdb\LINE00240"
inPoint = r"D:\TEMP\Output_20200520080600.gdb\StructureJoin"
testtype = "fromdisk"
if testtype == "fromdisk":
# Features are stored on disk
# Feature layers are created from layers on disk
print("Running disk workflow")
# Make feature layers
arcpy.MakeFeatureLayer_management(inLine, "LineFeature")
arcpy.MakeFeatureLayer_management(inPoint, "StructureFeature")
# Add join
arcpy.AddJoin_management("LineFeature", "OBJECTID", "StructureFeature", "ORIG_FID", "KEEP_COMMON")
# Calculate field
in_field = "Field_Name"
expression = '!StructureJoin.STRUCTURE_GUID!'
arcpy.CalculateField_management("LineFeature", in_field, expression, "PYTHON3")
elif testtype == "frommemory":
# Features are stored on disk
# Make copy of data in memory
# Feature layers are created from layers in memory
print("Running memory workflow")
# Copy features to memory
arcpy.CopyFeatures_management(inLine, r"memory\Lines")
arcpy.CopyFeatures_management(inPoint, r"memory\Structures")
# Make feature layers
arcpy.MakeFeatureLayer_management(r"memory\Lines", "LineFeature")
arcpy.MakeFeatureLayer_management(r"memory\Structures", "StructureFeature")
# Add join
arcpy.AddJoin_management("LineFeature", "OBJECTID", "StructureFeature", "ORIG_FID", "KEEP_COMMON")
# Calculate field
in_field = "Field_Name"
expression = '!Structures.STRUCTURE_GUID!'
arcpy.CalculateField_management("LineFeature", in_field, expression, "PYTHON3")
The resulting error is:
Running memory workflow
Traceback (most recent call last):
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\management.py", line 4527, in CalculateField
retval = convertArcObjectToPythonObject(gp.CalculateField_management(*gp_fixargs((in_table, field, expression, expression_type, code_block, field_type), True)))
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\geoprocessing\_base.py", line 511, in <lambda>
return lambda *args: val(*gp_fixargs(args, True))
arcgisscripting.ExecuteError: ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds.
Failed to execute (CalculateField).
Do I need to do something different when using the memory workspace?
Solved! Go to Solution.
I am a huge fan of Richard Fairhurst's approach described here: /blogs/richard_fairhurst/2014/11/08/turbo-charging-data-manipulation-with-python-cursors-and-diction... and use it often. You may want to give it a try.
I am a huge fan of Richard Fairhurst's approach described here: /blogs/richard_fairhurst/2014/11/08/turbo-charging-data-manipulation-with-python-cursors-and-diction... and use it often. You may want to give it a try.
Thank you for this link - I'm going to give the dictionary example a try and see if that works with the in-memory features. Thank you for the pointer!
It's really cool; no memory, no join, just get the data from the source to the target: I have used it in memory, but only because I was updating in memory tables from other in memory tables where the data eventually found it's way to a feature class on disk.
You could try utilising the arpy.env.scratchWorkspace e.g. output_path = os.path.join(arcpy.env.scratchWorkspace, output_name), it's also a useful workaround for GP tools which contain functions requiring physical outputs/copies, the server handles them in a pseudo in_memory capacity (as far as the user is concerned).
In the original version of this script I was writing everything out to disk first, and that worked fine, and I confirmed that the field calculation works that way. I'm trying to do as much in memory as possible to eliminate the disk I/O.
In-memory workspaces (memory and in_memory) are not full-fledged workspaces, some geodatabase features/functionality work, some don't. In this case, I do think it is a defect because I was able to use the older in_memory workspace in a standalone script and it works fine, no errors with expected results.
Seems like there have been other threads discussing some short comings of the memory workspace; I've never used it, just in_memory...
Using an example from the blog post that Joe shared I was able to get the calculation to work with the UpdateCursor in the memory workspace, so it looks like I'm good to proceed. I didn't test using the older in_memory workspace to see if my join + calculation method would work, but I'd rather get it to work with the newer workspace type.
I'm glad it worked for you; I can't tell you how many times I've used that approach!