Select to view content in your preferred language

CalculateGeometryAttributes produces None values in multiprocessing

416
10
Jump to solution
4 weeks ago
HieuTran12
New Contributor II

I was trying to perform a geo analysis and the function CalculateGeometryAttributes produced None values when I wrapped the script in multiprocessing, but it worked correctly in a for loop.

 

def process(fc):
        arcpy.env.workspace = "G:/usgrids2020/test1/"
        arcpy.env.overwriteOutput=True
        state = fc[34:-7]
        print(f"Processing for {state.upper()}")
        readTime = datetime.datetime.now()
        # arcpy.env.workspace = os.path.join(gdb_path, f"usa{state}.gdb")
        #Import the feature for boundary and water
        # fc = f"usa{state[3:5]}_joined"
        fcInMem = arcpy.management.CopyFeatures(fc, f"in_memory/usa{state}_fcInMem")
        water_fc = f"{fc[:-7]}_water_layer"
        waterInMem = arcpy.management.CopyFeatures(water_fc, f"in_memory/usa{state}_waterInMem")
        #Create the temporary id for joining fields later
        arcpy.AddField_management(fcInMem,'tempid','LONG')
        arcpy.CalculateField_management(fcInMem,"tempid",'!OBJECTID!','PYTHON')
        #Calculate the area(land+water) by GEODESIC method => Kytt recommended
        arcpy.management.AddField(fcInMem, "GEODESIC_AREA", "FLOAT")
        arcpy.management.CalculateGeometryAttributes(fcInMem, "GEODESIC_AREA AREA_GEODESIC", "", "SQUARE_METERS")        

This is what it generated:

(None, 'USA_100030117003015')
(None, 'USA_100030127001011')
(None, 'USA_100030151002006')
(None, 'USA_100030121001003')
(None, 'USA_100030112051009')
(None, 'USA_100030148102015')
(None, 'USA_100030112062022')
(None, 'USA_100030168063009')
(None, 'USA_100030166141010')
(None, 'USA_100030164041013')

 

0 Kudos
1 Solution

Accepted Solutions
HaydenWelch
Occasional Contributor II

Did you make sure that the feature units are meters? If so you can try this:

import arcpy
import datetime

def process(fc):
        arcpy.env.workspace = "G:/usgrids2020/test1/"
        arcpy.env.overwriteOutput=True
        state = fc[34:-7]
        print(f"Processing for {state.upper()}")
        readTime = datetime.datetime.now()
        # arcpy.env.workspace = os.path.join(gdb_path, f"usa{state}.gdb")
        #Import the feature for boundary and water
        # fc = f"usa{state[3:5]}_joined"
        fcInMem = arcpy.management.CopyFeatures(fc, f"in_memory/usa{state}_fcInMem")
        water_fc = f"{fc[:-7]}_water_layer"
        waterInMem = arcpy.management.CopyFeatures(water_fc, f"in_memory/usa{state}_waterInMem")
        #Create the temporary id for joining fields later
        arcpy.AddField_management(fcInMem,'tempid','LONG')
        arcpy.CalculateField_management(fcInMem,"tempid",'!OBJECTID!','PYTHON')
        #Calculate the area(land+water) by GEODESIC method => Kytt recommended
        arcpy.management.AddField(fcInMem, "GEODESIC_AREA", "FLOAT")
        with arcpy.da.UpdateCursor(fcInMem, ["SHAPE@", "GEODESIC_AREA"]) as cursor:
            for row in cursor:
                row = dict(zip(cursor.fields, row))
                row['GEODESIC_AREA'] = row['SHAPE@'].getArea('GEODESIC', 'SquareMeters')
                cursor.updateRow(row.values())
        #arcpy.management.CalculateGeometryAttributes(fcInMem, "GEODESIC_AREA AREA_GEODESIC", "", "SQUARE_METERS")

Which uses the Polygon.getArea() method. It pulls the whole shape object though so if you're doing this for millions of rows it can slow down a bit, but you'll get an accurate Geodesic area.

View solution in original post

0 Kudos
10 Replies
DanPatterson
MVP Esteemed Contributor

does the newer "memory" workspace behave differently than the "in_memory" workspace with multiprocessing?

I suspect a for loop would be better if using memory workspaces in any event


... sort of retired...
0 Kudos
HieuTran12
New Contributor II

I tried without the "in_memory" as well but the result remains the same. 

0 Kudos
JoshuaBixby
MVP Esteemed Contributor

Commented at python - ArcPy function produced None values in multiprocessing - GIS SE, it looks to be a defect with CalculateGeometryAttributes. I was able to create a minimum reproducible example and have submitted to Esri Support to log a defect.

HieuTran12
New Contributor II

Hi Joshua, have you heard anything from them yet? 

Apparently, I need that function in order to run to get some data from it. There is alternative solution that Hayden posted below but I couldn't use it for my case.

0 Kudos
JoshuaBixby
MVP Esteemed Contributor

Spent time working through the issue with Esri Support, and it is complex.  The issue has nothing to do with any specific type of workspace, nor anything with CalculateGeometryAttributes itself, it has to do with how Esri has structured their code for hundreds of Python-based tools like CalculateGeometryAttributes.  There are literally hundreds of tools that will all fail to run with Python multiprocessing.  The crazy part is, for most of the tools only a single line of code needs to be changed to make it work.

The question then becomes are these issues with documentation or the software, are these defects or enhancement requests, etc....  I haven't closed out the Esri Support case yet, but I suspect it will be a mixture of documentation defects and software enhancements that come out of it.

In the meantime, one can at least roll their own code with ArcPy DA cursors.

HaydenWelch
Occasional Contributor II

Since this seems to be a bug as per @JoshuaBixby's reply, you could try using an UpdateCursor to get the area for now:

import arcpy
import datetime

def process(fc):
        arcpy.env.workspace = "G:/usgrids2020/test1/"
        arcpy.env.overwriteOutput=True
        state = fc[34:-7]
        print(f"Processing for {state.upper()}")
        readTime = datetime.datetime.now()
        # arcpy.env.workspace = os.path.join(gdb_path, f"usa{state}.gdb")
        #Import the feature for boundary and water
        # fc = f"usa{state[3:5]}_joined"
        fcInMem = arcpy.management.CopyFeatures(fc, f"in_memory/usa{state}_fcInMem")
        water_fc = f"{fc[:-7]}_water_layer"
        waterInMem = arcpy.management.CopyFeatures(water_fc, f"in_memory/usa{state}_waterInMem")
        #Create the temporary id for joining fields later
        arcpy.AddField_management(fcInMem,'tempid','LONG')
        arcpy.CalculateField_management(fcInMem,"tempid",'!OBJECTID!','PYTHON')
        #Calculate the area(land+water) by GEODESIC method => Kytt recommended
        arcpy.management.AddField(fcInMem, "GEODESIC_AREA", "FLOAT")
        with arcpy.da.UpdateCursor(fcInMem, ["SHAPE@AREA", "GEODESIC_AREA"]) as cursor:
            for row in cursor:
                row = dict(zip(cursor.fields, row))
                row['GEODESIC_AREA'] = row['SHAPE@AREA'] # This is in feature units, so you will likely need the conversion factor from arcpy.LinearUnitConversionFactor
                cursor.updateRow(row.values())
        #arcpy.management.CalculateGeometryAttributes(fcInMem, "GEODESIC_AREA AREA_GEODESIC", "", "SQUARE_METERS")
0 Kudos
HieuTran12
New Contributor II

I tried your solution, but the results are very far off from the Geodesic method for area calculations.

HaydenWelch
Occasional Contributor II

Did you make sure that the feature units are meters? If so you can try this:

import arcpy
import datetime

def process(fc):
        arcpy.env.workspace = "G:/usgrids2020/test1/"
        arcpy.env.overwriteOutput=True
        state = fc[34:-7]
        print(f"Processing for {state.upper()}")
        readTime = datetime.datetime.now()
        # arcpy.env.workspace = os.path.join(gdb_path, f"usa{state}.gdb")
        #Import the feature for boundary and water
        # fc = f"usa{state[3:5]}_joined"
        fcInMem = arcpy.management.CopyFeatures(fc, f"in_memory/usa{state}_fcInMem")
        water_fc = f"{fc[:-7]}_water_layer"
        waterInMem = arcpy.management.CopyFeatures(water_fc, f"in_memory/usa{state}_waterInMem")
        #Create the temporary id for joining fields later
        arcpy.AddField_management(fcInMem,'tempid','LONG')
        arcpy.CalculateField_management(fcInMem,"tempid",'!OBJECTID!','PYTHON')
        #Calculate the area(land+water) by GEODESIC method => Kytt recommended
        arcpy.management.AddField(fcInMem, "GEODESIC_AREA", "FLOAT")
        with arcpy.da.UpdateCursor(fcInMem, ["SHAPE@", "GEODESIC_AREA"]) as cursor:
            for row in cursor:
                row = dict(zip(cursor.fields, row))
                row['GEODESIC_AREA'] = row['SHAPE@'].getArea('GEODESIC', 'SquareMeters')
                cursor.updateRow(row.values())
        #arcpy.management.CalculateGeometryAttributes(fcInMem, "GEODESIC_AREA AREA_GEODESIC", "", "SQUARE_METERS")

Which uses the Polygon.getArea() method. It pulls the whole shape object though so if you're doing this for millions of rows it can slow down a bit, but you'll get an accurate Geodesic area.

0 Kudos
HieuTran12
New Contributor II

Wow, this could be an alternate method of CalculateGeometryAttributes. Or does it have any tradeoffs?