topic Re: Copy and Project Features Slowing Down Script in Python Window in ArcGIS Pro Questions

Copy and Project Features Slowing Down Script in Python Window

deleted-user-1_r2dgYuILKY — Thu, 28 Mar 2019 15:46:07 GMT

I have a script that copies thousands of features, performs analysis and then deletes the copies. When I run this script in the Python window in ArcGIS Pro 2.3.1, each feature is copied and displayed in the TOC (ex. feature_new), and then is deleted. However, each time this happens, another copy with the original feature name and a number (ex. feature_old26) is created in the TOC, and remains after the script moves on to the next feature. So I have thousands of these in the TOC (see photo). If I use arcpy.env.addOutputsToMap = False in the script, the script immediately fails on copy features. ERROR 000732: Input Features: Dataset feature_old26 does not exist or is not supported. I know that this particular bug was logged almost a year ago (BUG-000116163), but I can't find any information about it.

The issue I have with this is that it's dramatically slowing down the Python script execution over time as hundreds or thousands of features build up in the TOC. I would run it in standalone IDLE, but for some reason it's faster in the Python window at first.

Re: Copy Features Slowing Down Script in Python Window

DrewFlater — Thu, 28 Mar 2019 18:11:05 GMT

Hi,

The bug you linked, BUG-000116163, is not fixed yet. It deals specifically with arcpy.MakeFeatureLayer_management() when arcpy.env.addOutputToMap = False.

If your script is only doing a CopyFeatures then it might be a different issue that the script is failing. You do say after the copy the script "performs analysis" so I guess this may include arcpy.MakeFeatureLayer_management? You might be able to rewrite some steps in your script, as Make Feature Layer is no longer required as a pre-requisite for many tools like Add Join, Select Layer By Attribute, etc, as of Pro 2.2. https://pro.arcgis.com/en/pro-app/get-started/what-s-new-in-arcgis-pro-2-2.htm#ESRI_SECTION1_68F7BAF048AE4EF096C802EB7FD68817 (see the second bullet point)

Re: Copy Features Slowing Down Script in Python Window

MatthewDriscoll — Thu, 28 Mar 2019 18:18:04 GMT

You also might try creating the features in memory, there would be no need to delete them.

Re: Copy Features Slowing Down Script in Python Window

deleted-user-1_r2dgYuILKY — Thu, 28 Mar 2019 19:31:28 GMT

No, MakeFeatureLayer is not part of the script. It's peeling off and copying individual rooms from buildings, generating a tessellation for the room, select by location to count the hexagons and then moving on to the next room. All of those interim steps are deleted each time. The problem is, every time there's a copy of a room, another copy of the main feature class (all the buildings) is created in the TOC with a number at the end. That's filling up the TOC, and as far as I can tell, slowing the script down. When it starts, it's taking 40-45 seconds per room. After several hours it's up to around 3 minutes. Has anyone seen that happen before? It's very strange. Would creating the copies in memory solve this issue?

Re: Copy Features Slowing Down Script in Python Window

JoshuaBixby — Thu, 28 Mar 2019 19:36:24 GMT

Please provide the code that is removing the layers from the TOC, or is the user doing it manually in the GUI?

Re: Copy Features Slowing Down Script in Python Window

deleted-user-1_r2dgYuILKY — Thu, 28 Mar 2019 20:29:01 GMT

It's not removing them from the TOC, but the scratch.gdb. Pro is adding these copied layers to the TOC during the analysis, and then they are removed once they're deleted from the geodatabase. I'd rather they aren't added to the map at all, but due to that bug, I can't control it.

To reiterate: I have one feature class with nearly 100 buildings that's been converted from CAD. The script goes through each building, copies out each room, runs analysis, and moves on to the next room. What's building up in the TOC is unexplained copies of the original feature class (see photo above) that are happening in tandem with the room feature copy. That is a feature class with 12400+ rows copied over and over again in the TOC. I believe it's clogging up the memory and slowing the Python script way down.

Update: I believe this is a problem with the Project tool. I just tried to write the interim steps to memory and got this ERROR 000944: Output feature class cannot be in the in_memory workspace. Failed to execute (Project).

Here is a better illustration of what's happening. The bottom feature is the original, the top feature is a single room that's been copied (based on an attribute selection from the main feature) and projected to the coordinate system based on the central meridian of the room. The middle feature is the copy of the original that remains in the TOC. Its source is identical to the main feature class, so I believe it's just being copied in memory by Pro.

Re: Copy and Project Features Slowing Down Script in Python Window

MatthewDriscoll — Mon, 01 Apr 2019 17:04:23 GMT

I believe you still need to make a feature layer from your feature class if you are going to run it in_memory.

Re: Copy and Project Features Slowing Down Script in Python Window

deleted-user-1_r2dgYuILKY — Mon, 01 Apr 2019 17:12:30 GMT

OK, but I still think there there is a bug here. There should not be extra copies of the original feature class clogging up the TOC.

Re: Copy and Project Features Slowing Down Script in Python Window

MatthewDriscoll — Mon, 01 Apr 2019 18:04:55 GMT

Hard to say without seeing the code. You may not be finishing the loop cleanly before moving onto the next one. If it is clean I recently learned that MapView.Redraw was introduced in 2.2, you might give that a try before moving onto the next one.

Re: Copy and Project Features Slowing Down Script in Python Window

JoshuaBixby — Mon, 01 Apr 2019 21:28:51 GMT

How or where are you executing this script? From the behavior you are seeing, I am guessing from the interactive Python window and not a script tool. Also, I agree with Matthew Driscoll‌, we really need to see some actual code to give more substantive feedback.

Re: Copy and Project Features Slowing Down Script in Python Window

deleted-user-1_r2dgYuILKY — Sun, 12 Dec 2021 03:58:53 GMT

Yes, I was executing it in the Python window because it was faster, for a while. I just went back to running it in IDLE.

Here is a truncated version of the script:

import arcpy
import os, sys
import time
from datetime import date, timedelta

output_gdb = r'M:\\PROJECTS\\Projects_2018-19\\Map Requests\\Room Capacity Analysis\\room_capacity.gdb\\'
scratch_gdb = r'M:\\PROJECTS\\Projects_2018-19\\Map Requests\\Room Capacity Analysis\\scratch.gdb\\'
memory_hole = r'in_memory\\'

arcpy.env.workspace = scratch_gdb
arcpy.env.overwriteOutput = True
arcpy.env.outputZFlag = "Disabled"
#arcpy.env.addOutputsToMap = False

fc = output_gdb + r'Rooms_w_SpaceID_AllSites_Proj'
site = ""
room = ""
area_per = ""


exclude_list = ["BUILDING_A","BUILDING_B"]

sql_string = "WHERE UsableSF > 300 "

print("Starting at " + str(datetime.datetime.now()))

with arcpy.da.SearchCursor(fc, '*' ,sql_clause=(None, sql_string)) as cursor:

    for row in cursor:
        
        OID_field = arcpy.Describe(fc).OIDFieldName
        site = row[8]
        room_number = row[9]
        usable_square_footage = row[7]
        space_class = row[11]
        spatial_ref = row[21]
  
        if site in exclude_list:
            continue
        
        else:
            start_time = time.time()
    
            site_no_space = site.replace(' ','_')
            room_no_space = room_number.replace(' ','_')
            site_no_dash = site_no_space.replace('-','_')
            site_no_dot  = site_no_dash.replace('.','_')
            room_no_dash = room_no_space.replace('-','')
            room_no_dot = room_no_dash.replace('.','')
            room_feature = memory_hole + site_no_dot.lower() + '_room_' + room_no_dot
            room_feature_proj = site_no_dot.lower() + "_room_" + room_no_dot + "_proj"
            
            where = '{0} = {1}'.format(arcpy.AddFieldDelimiters(fc,OID_field),row[0]) 
            selection = arcpy.SelectLayerByAttribute_management(fc,"NEW_SELECTION",where) # select the current room feature  
            arcpy.CopyFeatures_management(selection,room_feature) # output room to new feature class
            arcpy.Project_management(room_feature, scratch_gdb + room_feature_proj, spatial_ref)

            ## get room extent for grid
            description = arcpy.Describe(room_feature_proj)
            extent = description.extent
          
            thex_grid = memory_hole + room_feature_proj + "_thex_grid"

            
            area_per = "30 SquareFeet"
  
            footage = int(area_per[:3])

            ## generate transverse hexagon grid polygon
            arcpy.management.GenerateTessellation(thex_grid, extent, "TRANSVERSE_HEXAGON", area_per, spatial_ref) 
            ## Search for grid squares that fall completely within the room
            new_selection = arcpy.management.SelectLayerByLocation(thex_grid, "COMPLETELY_WITHIN", room_feature_proj, None, "NEW_SELECTION", "NOT_INVERT")

            count = arcpy.GetCount_management(new_selection)
             

            arcpy.AddField_management(room_feature_proj, "SQFT", "SHORT", 3, "", "", "SQFTPerStudent", "NULLABLE", "REQUIRED")
            arcpy.CalculateField_management(room_feature_proj, "SQFT", footage, "PYTHON_9.3")
            arcpy.AddField_management(room_feature_proj, "Count", "SHORT", 3, "", "", "OperationalCapacity", "NULLABLE", "REQUIRED")
            arcpy.CalculateField_management(room_feature_proj, "Count", count, "PYTHON_9.3")

            elapsed_time_secs = time.time() - start_time

            msg = "Execution time: %s" % timedelta(seconds=round(elapsed_time_secs))
            print(msg)

            arcpy.Append_management(room_feature_proj, output_gdb + "\\rooms_with_operational_capacity","", "","","")

del cursor
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Copy and Project Features Slowing Down Script in Python Window

JoshuaBixby — Tue, 02 Apr 2019 17:41:59 GMT

Does the script run as expected from IDLE? I assume so since your issue seems tied to TOC, but I hate assuming.

Re: Copy and Project Features Slowing Down Script in Python Window

deleted-user-1_r2dgYuILKY — Tue, 02 Apr 2019 17:58:48 GMT

It does run fine in IDLE, yes.

Re: Copy and Project Features Slowing Down Script in Python Window

MatthewDriscoll — Tue, 02 Apr 2019 19:08:44 GMT

Nothing really sticking out here for the reason of the behavior. I really do think it is something in your code and not a bug for the following reason:

The original example screen shot shows that a copy is getting created a lot but only every other third ends up in the table of contents. So it seems like it still creates a copy when it is in your exclude_list (continue) but does not go into your TOC.

I do agree that it creating a new copy of the original on every loop and some of them getting added to the TOC is a major factor to slowing it down.

Re: Copy and Project Features Slowing Down Script in Python Window

deleted-user-1_r2dgYuILKY — Tue, 02 Apr 2019 19:27:56 GMT

That's not what is happening at all. The exclude list is buildings (sites) from within the single feature class (Rooms_w_Space_ID_AllSites_Proj), which is nearly 100 buildings. All this list does is specify which buildings to skip over, it has nothing to do with the room copies. I just used generic names in the example, but there are over a dozen buildings excluded. Nowhere in the script do I specify a copy of the main feature class. This is happening inside Pro in the TOC when this script is run from the Python window.

I'll explain again: each room from each building is selected from the main feature class (Rooms_w_Space_ID_AllSites_Proj). This selection is copied as a feature, projected, analyzed and then all these steps are deleted. Next room, repeat. What's building up in the TOC are numbered copies of the main feature class (which shouldn't be happening at all). It's happening every single time that a room from the feature class is copied and projected to the central meridian of the room. There is something going on with the project tool in Pro. The photo above just shows the results of one iteration of this. There are thousands of these copies building up in the TOC as the script runs (see photo from my initial post).

Re: Copy and Project Features Slowing Down Script in Python Window

MatthewDriscoll — Sun, 12 Dec 2021 03:58:56 GMT

What I was pointing out was that if you look at the numbers, on the original photo, they are not +1, this shows that the copies of the main feature classes are getting copied more than are getting put on the TOC. I simplified the code and ran it and was not able to reproduce this problem.

Also, after reading the Project documentation I found I steered you down a wrong path, you cannot use the in_memory for that particular tool. Which is why you got that error.

import arcpy
arcpy.env.overwriteOutput = True
output_gdb = r"C:\\Users\\mdriscoll\\Desktop\\New Folder\\abc\\Default.gdb\\"
exclude_list = ["A","E"]
fc = output_gdb + "pnt"

with arcpy.da.SearchCursor(fc,"*") as cursor:
    copy = output_gdb + "copy"
    outCS = arcpy.SpatialReference('NAD 1983 UTM Zone 11N')
    outputCS = output_gdb + "copy_proj"

    for row in cursor:
        field = row[2]
        if field in exclude_list:
            continue
        else:
            arcpy.CopyFeatures_management(fc,copy)
            arcpy.Project_management(copy, outputCS, outCS)
            print(field)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Copy and Project Features Slowing Down Script in Python Window

deleted-user-1_r2dgYuILKY — Tue, 02 Apr 2019 20:31:09 GMT

OK, but how can I simplify my script to avoid this? I see you're not selecting a layer by attribute before your copy and project steps.

Re: Copy and Project Features Slowing Down Script in Python Window

MatthewDriscoll — Tue, 02 Apr 2019 20:42:21 GMT

I was just testing the Copy and Project to show that you are perhaps going down the wrong path. I did not see anything that stuck out. If I were you I might comment out as much as you can and maybe test it line by line until the behavior is reproduced to find your smoking gun.

Re: Copy and Project Features Slowing Down Script in Python Window

JoshuaBixby — Tue, 02 Apr 2019 20:46:29 GMT

What you are experiencing is a result of using the legacy "in_memory" space versus the new "memory" workspace in Pro: Write geoprocessing output to memory—ArcGIS Pro | ArcGIS Desktop

in_memory datasets cannot be displayed on a map. If you run a geoprocessing tool from the Geoprocessing pane or Python window and write the output dataset to in_memory, after processing, the output dataset will be copied to the project geodatabase and that geodatabase dataset will be added to the map. Writing to the project geodatabase does not occur when in_memory datasets are intermediate and are not added to a map.

I suggest you try using the new memory workspace.