dylanh0

Basic Geoprocessing in Pro is SLOW

Blog Post created by dylanh0 on Apr 10, 2019

I'll admit I have been a hold out switching to Pro since it first came out, but not for lack of trying. I periodically have pushed myself to learn it and more so to like it, and it is growing on me. It just seems like every time I try and use it for something, one thing or another does not work, and I either revert back to ArcMap or spend much more time than I would have on the task. 

 

I became motivated to start using Python 3 and creating custom script tools for Pro, and recently had a project come up that warranted creating a tool. The purpose is to count residential and commercial structures within a given polygon area. The steps are fairly simple: first select points within the given polygon of interest, then select parcels from that point selection, then query the parcel selection, then add a field from the resulting selection. Simple right? 

 

It is simple to do manually, but cumbersome since I have 20+ polygons to do this for. Initially I wrote the tool in python 3 and made a custom script tool with a single parameter, the polygon area of interest. The tool ran and seemed to take an unexpected amount of time - longer than it would have taken me to manually perform these selections. I decided to do an experiment...

 

I copied the code from Spyder into PyScripter, saving it as a python 2 script, opened up good 'ol ArcMap, added a custom script tool with the same single parameter (changed nothing about the underlying code) and ran it. The results were troubling. What took Pro 16 minutes to run, ArcMap accomplished in 2. That's 8 times slower for all you math whizzes. I ran it over and over in both programs using polygons of various sizes, and ArcMap's time was right around 2 minutes consistently, while Pro's times ranged from 10 to 16. 

 

I remember when Pro first came out what got me excited was its a 64-bit multi-threaded application yada yada lightening fast... Well this kind of performance for this basic of a task is unacceptable for an application that is supposedly the GIS professional's future. 

 

# Import modules
import arcpy, os, sys
arcpy.env.overwriteOutput = True

# Connect to prod and get address points and parcels
sde_prod = "C:\\Users\\dylanh\\AppData\\Roaming\\Esri\\ArcGISPro\\Favorites\\gjgisprod.sde"
address_points = os.path.join(sde_prod, "gjgisprod.DBO.publicsafety_AEGIS",
                              "gjgisprod.DBO.NWSAegisGISDB_Addresses")
parcels = os.path.join(sde_prod, "gjgisprod.DBO.Parcel")

# Get user input for polygon layer
polygon_input = arcpy.GetParameterAsText(0)

# Ensure all layers exist
data = [address_points, parcels, polygon_input]
for fc in data:
    if arcpy.Exists(fc):
        arcpy.AddMessage(fc + " Exists!")
        print(fc + " Exists!")
    else:
        arcpy.AddMessage(fc + " Not Found, Exiting Script...")
        print(fc + " Not found!")
        sys.exit()

# Create feature layers for address points and parcels
arcpy.MakeFeatureLayer_management(address_points, "address_lyr")
arcpy.SelectLayerByLocation_management("address_lyr", "INTERSECT",
                                       polygon_input, "", "NEW_SELECTION")
arcpy.MakeFeatureLayer_management(parcels, "parcels_lyr")
arcpy.SelectLayerByLocation_management("parcels_lyr", "INTERSECT",
                                       "address_lyr", "", "NEW_SELECTION")
arcpy.AddMessage("Selected addresses: "
                 + str(arcpy.GetCount_management("address_lyr")))
arcpy.AddMessage("Selected parcels: "
                 + str(arcpy.GetCount_management("parcels_lyr")))

# Set up where clauses for residential and commercial property type
proptyp_field = "PROPTYPE"
res_list = ('Agricultural', 'Condo', 'Duplex/Triplex', 'Multi-Fam 4-8',
            'Multi 9 - Up', 'Residential', 'Townhouse')
com_list = ('Commercial', 'Exempt', 'Industrial', 'Percent Taxable / Ex',
            'State Assessed')
where_clause_res = """{0} IN {1}""".format(
        arcpy.AddFieldDelimiters(sde_prod, proptyp_field), res_list)
where_clause_com = """{0} IN {1}""".format(
        arcpy.AddFieldDelimiters(sde_prod, proptyp_field), com_list)

# Create feature layers for residential and commercial parcels from selection
arcpy.MakeFeatureLayer_management("parcels_lyr", "res_parcels",
                                  where_clause_res)
arcpy.MakeFeatureLayer_management("parcels_lyr", "com_parcels",
                                  where_clause_com)
arcpy.AddMessage("Residential parcels: "
                 + str(arcpy.GetCount_management("res_parcels")))
arcpy.AddMessage("Commercial parcels: "
                 + str(arcpy.GetCount_management("com_parcels")))

# Count number of buildings with cursor
building_field = ['TOTNOBLDGS']
residential = 0
commercial = 0

with arcpy.da.SearchCursor("res_parcels", building_field) as cursor:
    for row in cursor:
        if row[0]:
            residential = residential + row[0]

with arcpy.da.SearchCursor("com_parcels", building_field) as cursor:
    for row in cursor:
        if row[0]:
            commercial = commercial + row[0]

arcpy.AddMessage("Residential structures: " + str(residential))
arcpy.AddMessage("Commercial structures: " + str(commercial))

Outcomes