Copy and Project Features Slowing Down Script in Python Window

deleted-user-1_r2dgYuILKY · ‎03-28-2019

I have a script that copies thousands of features, performs analysis and then deletes the copies. When I run this script in the Python window in ArcGIS Pro 2.3.1, each feature is copied and displayed in the TOC (ex. feature_new), and then is deleted. However, each time this happens, another copy with the original feature name and a number (ex. feature_old26) is created in the TOC, and remains after the script moves on to the next feature. So I have thousands of these in the TOC (see photo). If I use arcpy.env.addOutputsToMap = False in the script, the script immediately fails on copy features. ERROR 000732: Input Features: Dataset feature_old26 does not exist or is not supported. I know that this particular bug was logged almost a year ago (BUG-000116163), but I can't find any information about it.

The issue I have with this is that it's dramatically slowing down the Python script execution over time as hundreds or thousands of features build up in the TOC. I would run it in standalone IDLE, but for some reason it's faster in the Python window at first.

deleted-user-1_r2dgYuILKY · ‎04-02-2019

Yes, I was executing it in the Python window because it was faster, for a while. I just went back to running it in IDLE.

Here is a truncated version of the script:

import arcpy
import os, sys
import time
from datetime import date, timedelta

output_gdb = r'M:\\PROJECTS\\Projects_2018-19\\Map Requests\\Room Capacity Analysis\\room_capacity.gdb\\'
scratch_gdb = r'M:\\PROJECTS\\Projects_2018-19\\Map Requests\\Room Capacity Analysis\\scratch.gdb\\'
memory_hole = r'in_memory\\'

arcpy.env.workspace = scratch_gdb
arcpy.env.overwriteOutput = True
arcpy.env.outputZFlag = "Disabled"
#arcpy.env.addOutputsToMap = False

fc = output_gdb + r'Rooms_w_SpaceID_AllSites_Proj'
site = ""
room = ""
area_per = ""


exclude_list = ["BUILDING_A","BUILDING_B"]

sql_string = "WHERE UsableSF > 300 "

print("Starting at " + str(datetime.datetime.now()))

with arcpy.da.SearchCursor(fc, '*' ,sql_clause=(None, sql_string)) as cursor:

    for row in cursor:
        
        OID_field = arcpy.Describe(fc).OIDFieldName
        site = row[8]
        room_number = row[9]
        usable_square_footage = row[7]
        space_class = row[11]
        spatial_ref = row[21]
  
        if site in exclude_list:
            continue
        
        else:
            start_time = time.time()
    
            site_no_space = site.replace(' ','_')
            room_no_space = room_number.replace(' ','_')
            site_no_dash = site_no_space.replace('-','_')
            site_no_dot  = site_no_dash.replace('.','_')
            room_no_dash = room_no_space.replace('-','')
            room_no_dot = room_no_dash.replace('.','')
            room_feature = memory_hole + site_no_dot.lower() + '_room_' + room_no_dot
            room_feature_proj = site_no_dot.lower() + "_room_" + room_no_dot + "_proj"
            
            where = '{0} = {1}'.format(arcpy.AddFieldDelimiters(fc,OID_field),row[0]) 
            selection = arcpy.SelectLayerByAttribute_management(fc,"NEW_SELECTION",where) # select the current room feature  
            arcpy.CopyFeatures_management(selection,room_feature) # output room to new feature class
            arcpy.Project_management(room_feature, scratch_gdb + room_feature_proj, spatial_ref)

            ## get room extent for grid
            description = arcpy.Describe(room_feature_proj)
            extent = description.extent
          
            thex_grid = memory_hole + room_feature_proj + "_thex_grid"

            
            area_per = "30 SquareFeet"
  
            footage = int(area_per[:3])

            ## generate transverse hexagon grid polygon
            arcpy.management.GenerateTessellation(thex_grid, extent, "TRANSVERSE_HEXAGON", area_per, spatial_ref) 
            ## Search for grid squares that fall completely within the room
            new_selection = arcpy.management.SelectLayerByLocation(thex_grid, "COMPLETELY_WITHIN", room_feature_proj, None, "NEW_SELECTION", "NOT_INVERT")

            count = arcpy.GetCount_management(new_selection)
             

            arcpy.AddField_management(room_feature_proj, "SQFT", "SHORT", 3, "", "", "SQFTPerStudent", "NULLABLE", "REQUIRED")
            arcpy.CalculateField_management(room_feature_proj, "SQFT", footage, "PYTHON_9.3")
            arcpy.AddField_management(room_feature_proj, "Count", "SHORT", 3, "", "", "OperationalCapacity", "NULLABLE", "REQUIRED")
            arcpy.CalculateField_management(room_feature_proj, "Count", count, "PYTHON_9.3")

            elapsed_time_secs = time.time() - start_time

            msg = "Execution time: %s" % timedelta(seconds=round(elapsed_time_secs))
            print(msg)

            arcpy.Append_management(room_feature_proj, output_gdb + "\\rooms_with_operational_capacity","", "","","")

del cursor
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

JoshuaBixby · ‎04-02-2019

Does the script run as expected from IDLE? I assume so since your issue seems tied to TOC, but I hate assuming.

deleted-user-1_r2dgYuILKY · ‎04-02-2019

It does run fine in IDLE, yes.

JoshuaBixby · ‎04-02-2019

What you are experiencing is a result of using the legacy "in_memory" space versus the new "memory" workspace in Pro: Write geoprocessing output to memory—ArcGIS Pro | ArcGIS Desktop

in_memory datasets cannot be displayed on a map. If you run a geoprocessing tool from the Geoprocessing pane or Python window and write the output dataset to in_memory, after processing, the output dataset will be copied to the project geodatabase and that geodatabase dataset will be added to the map. Writing to the project geodatabase does not occur when in_memory datasets are intermediate and are not added to a map.

I suggest you try using the new memory workspace.

MatthewDriscoll · ‎04-02-2019

Nothing really sticking out here for the reason of the behavior. I really do think it is something in your code and not a bug for the following reason:

The original example screen shot shows that a copy is getting created a lot but only every other third ends up in the table of contents. So it seems like it still creates a copy when it is in your exclude_list (continue) but does not go into your TOC.

I do agree that it creating a new copy of the original on every loop and some of them getting added to the TOC is a major factor to slowing it down.

deleted-user-1_r2dgYuILKY · ‎04-02-2019

That's not what is happening at all. The exclude list is buildings (sites) from within the single feature class (Rooms_w_Space_ID_AllSites_Proj), which is nearly 100 buildings. All this list does is specify which buildings to skip over, it has nothing to do with the room copies. I just used generic names in the example, but there are over a dozen buildings excluded. Nowhere in the script do I specify a copy of the main feature class. This is happening inside Pro in the TOC when this script is run from the Python window.

I'll explain again: each room from each building is selected from the main feature class (Rooms_w_Space_ID_AllSites_Proj). This selection is copied as a feature, projected, analyzed and then all these steps are deleted. Next room, repeat. What's building up in the TOC are numbered copies of the main feature class (which shouldn't be happening at all). It's happening every single time that a room from the feature class is copied and projected to the central meridian of the room. There is something going on with the project tool in Pro. The photo above just shows the results of one iteration of this. There are thousands of these copies building up in the TOC as the script runs (see photo from my initial post).

MatthewDriscoll · ‎04-02-2019

What I was pointing out was that if you look at the numbers, on the original photo, they are not +1, this shows that the copies of the main feature classes are getting copied more than are getting put on the TOC. I simplified the code and ran it and was not able to reproduce this problem.

Also, after reading the Project documentation I found I steered you down a wrong path, you cannot use the in_memory for that particular tool. Which is why you got that error.

import arcpy
arcpy.env.overwriteOutput = True
output_gdb = r"C:\\Users\\mdriscoll\\Desktop\\New Folder\\abc\\Default.gdb\\"
exclude_list = ["A","E"]
fc = output_gdb + "pnt"

with arcpy.da.SearchCursor(fc,"*") as cursor:
    copy = output_gdb + "copy"
    outCS = arcpy.SpatialReference('NAD 1983 UTM Zone 11N')
    outputCS = output_gdb + "copy_proj"

    for row in cursor:
        field = row[2]
        if field in exclude_list:
            continue
        else:
            arcpy.CopyFeatures_management(fc,copy)
            arcpy.Project_management(copy, outputCS, outCS)
            print(field)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

deleted-user-1_r2dgYuILKY · ‎04-02-2019

OK, but how can I simplify my script to avoid this? I see you're not selecting a layer by attribute before your copy and project steps.

MatthewDriscoll · ‎04-02-2019

I was just testing the Copy and Project to show that you are perhaps going down the wrong path. I did not see anything that stuck out. If I were you I might comment out as much as you can and maybe test it line by line until the behavior is reproduced to find your smoking gun.