topic Re: CalculateGeometryAttributes produces None values in multiprocessing in Python Questions

CalculateGeometryAttributes produces None values in multiprocessing

HieuTran12 — Tue, 18 Jun 2024 23:37:03 GMT

I was trying to perform a geo analysis and the function CalculateGeometryAttributes produced None values when I wrapped the script in multiprocessing, but it worked correctly in a for loop.

def process(fc): arcpy.env.workspace = "G:/usgrids2020/test1/" arcpy.env.overwriteOutput=True state = fc[34:-7] print(f"Processing for {state.upper()}") readTime = datetime.datetime.now() # arcpy.env.workspace = os.path.join(gdb_path, f"usa{state}.gdb") #Import the feature for boundary and water # fc = f"usa{state[3:5]}_joined" fcInMem = arcpy.management.CopyFeatures(fc, f"in_memory/usa{state}_fcInMem") water_fc = f"{fc[:-7]}_water_layer" waterInMem = arcpy.management.CopyFeatures(water_fc, f"in_memory/usa{state}_waterInMem") #Create the temporary id for joining fields later arcpy.AddField_management(fcInMem,'tempid','LONG') arcpy.CalculateField_management(fcInMem,"tempid",'!OBJECTID!','PYTHON') #Calculate the area(land+water) by GEODESIC method => Kytt recommended arcpy.management.AddField(fcInMem, "GEODESIC_AREA", "FLOAT") arcpy.management.CalculateGeometryAttributes(fcInMem, "GEODESIC_AREA AREA_GEODESIC", "", "SQUARE_METERS")

This is what it generated:

(None, 'USA_100030117003015') (None, 'USA_100030127001011') (None, 'USA_100030151002006') (None, 'USA_100030121001003') (None, 'USA_100030112051009') (None, 'USA_100030148102015') (None, 'USA_100030112062022') (None, 'USA_100030168063009') (None, 'USA_100030166141010') (None, 'USA_100030164041013')

Re: CalculateGeometryAttributes produces None values in multiprocessing

DanPatterson — Wed, 19 Jun 2024 01:06:57 GMT

does the newer "memory" workspace behave differently than the "in_memory" workspace with multiprocessing?

I suspect a for loop would be better if using memory workspaces in any event

Re: CalculateGeometryAttributes produces None values in multiprocessing

HieuTran12 — Wed, 19 Jun 2024 01:22:05 GMT

I tried without the "in_memory" as well but the result remains the same.

Re: CalculateGeometryAttributes produces None values in multiprocessing

JoshuaBixby — Wed, 19 Jun 2024 13:55:59 GMT

Commented at python - ArcPy function produced None values in multiprocessing - GIS SE, it looks to be a defect with CalculateGeometryAttributes. I was able to create a minimum reproducible example and have submitted to Esri Support to log a defect.

Re: CalculateGeometryAttributes produces None values in multiprocessing

HaydenWelch — Thu, 20 Jun 2024 14:45:27 GMT

Since this seems to be a bug as per @JoshuaBixby's reply, you could try using an UpdateCursor to get the area for now:

import arcpy import datetime def process(fc): arcpy.env.workspace = "G:/usgrids2020/test1/" arcpy.env.overwriteOutput=True state = fc[34:-7] print(f"Processing for {state.upper()}") readTime = datetime.datetime.now() # arcpy.env.workspace = os.path.join(gdb_path, f"usa{state}.gdb") #Import the feature for boundary and water # fc = f"usa{state[3:5]}_joined" fcInMem = arcpy.management.CopyFeatures(fc, f"in_memory/usa{state}_fcInMem") water_fc = f"{fc[:-7]}_water_layer" waterInMem = arcpy.management.CopyFeatures(water_fc, f"in_memory/usa{state}_waterInMem") #Create the temporary id for joining fields later arcpy.AddField_management(fcInMem,'tempid','LONG') arcpy.CalculateField_management(fcInMem,"tempid",'!OBJECTID!','PYTHON') #Calculate the area(land+water) by GEODESIC method => Kytt recommended arcpy.management.AddField(fcInMem, "GEODESIC_AREA", "FLOAT") with arcpy.da.UpdateCursor(fcInMem, ["SHAPE@AREA", "GEODESIC_AREA"]) as cursor: for row in cursor: row = dict(zip(cursor.fields, row)) row['GEODESIC_AREA'] = row['SHAPE@AREA'] # This is in feature units, so you will likely need the conversion factor from arcpy.LinearUnitConversionFactor cursor.updateRow(row.values()) #arcpy.management.CalculateGeometryAttributes(fcInMem, "GEODESIC_AREA AREA_GEODESIC", "", "SQUARE_METERS")

Re: CalculateGeometryAttributes produces None values in multiprocessing

HieuTran12 — Thu, 20 Jun 2024 17:55:30 GMT

I tried your solution, but the results are very far off from the Geodesic method for area calculations.

Re: CalculateGeometryAttributes produces None values in multiprocessing

HaydenWelch — Thu, 20 Jun 2024 19:19:29 GMT

Did you make sure that the feature units are meters? If so you can try this:

import arcpy import datetime def process(fc): arcpy.env.workspace = "G:/usgrids2020/test1/" arcpy.env.overwriteOutput=True state = fc[34:-7] print(f"Processing for {state.upper()}") readTime = datetime.datetime.now() # arcpy.env.workspace = os.path.join(gdb_path, f"usa{state}.gdb") #Import the feature for boundary and water # fc = f"usa{state[3:5]}_joined" fcInMem = arcpy.management.CopyFeatures(fc, f"in_memory/usa{state}_fcInMem") water_fc = f"{fc[:-7]}_water_layer" waterInMem = arcpy.management.CopyFeatures(water_fc, f"in_memory/usa{state}_waterInMem") #Create the temporary id for joining fields later arcpy.AddField_management(fcInMem,'tempid','LONG') arcpy.CalculateField_management(fcInMem,"tempid",'!OBJECTID!','PYTHON') #Calculate the area(land+water) by GEODESIC method => Kytt recommended arcpy.management.AddField(fcInMem, "GEODESIC_AREA", "FLOAT") with arcpy.da.UpdateCursor(fcInMem, ["SHAPE@", "GEODESIC_AREA"]) as cursor: for row in cursor: row = dict(zip(cursor.fields, row)) row['GEODESIC_AREA'] = row['SHAPE@'].getArea('GEODESIC', 'SquareMeters') cursor.updateRow(row.values()) #arcpy.management.CalculateGeometryAttributes(fcInMem, "GEODESIC_AREA AREA_GEODESIC", "", "SQUARE_METERS")

Which uses the Polygon.getArea() method. It pulls the whole shape object though so if you're doing this for millions of rows it can slow down a bit, but you'll get an accurate Geodesic area.

Re: CalculateGeometryAttributes produces None values in multiprocessing

HieuTran12 — Thu, 20 Jun 2024 20:33:20 GMT

Wow, this could be an alternate method of CalculateGeometryAttributes. Or does it have any tradeoffs?

Re: CalculateGeometryAttributes produces None values in multiprocessing

HaydenWelch — Thu, 20 Jun 2024 23:10:29 GMT

The tradeoff is that you have to use an arcpy cursor object, but I do most of my scripting with cursors. Even wrote some wrapper classes that allow you to use cursor objects safely using python syntax (see my post in Python Ideas).

Cursors are great because they gave you really granular access to the feature data. They are still Python objects though and don't have all the horsepower of the base Arc functions that are implemented in C.

For use cases like this though, and basic CRUD ops on a dataset they're great. Especially if you use the where_clause and spatial_filter parameters properly.

Re: CalculateGeometryAttributes produces None values in multiprocessing

HieuTran12 — Mon, 01 Jul 2024 20:49:53 GMT

Hi Joshua, have you heard anything from them yet?

Apparently, I need that function in order to run to get some data from it. There is alternative solution that Hayden posted below but I couldn't use it for my case.

Re: CalculateGeometryAttributes produces None values in multiprocessing

JoshuaBixby — Mon, 01 Jul 2024 21:46:20 GMT

Spent time working through the issue with Esri Support, and it is complex. The issue has nothing to do with any specific type of workspace, nor anything with CalculateGeometryAttributes itself, it has to do with how Esri has structured their code for hundreds of Python-based tools like CalculateGeometryAttributes. There are literally hundreds of tools that will all fail to run with Python multiprocessing. The crazy part is, for most of the tools only a single line of code needs to be changed to make it work.

The question then becomes are these issues with documentation or the software, are these defects or enhancement requests, etc.... I haven't closed out the Esri Support case yet, but I suspect it will be a mixture of documentation defects and software enhancements that come out of it.

In the meantime, one can at least roll their own code with ArcPy DA cursors.