AnsweredAssumed Answered

Copy and Project Features Slowing Down Script in Python Window

Question asked by LeviCecil on Mar 28, 2019
Latest reply on Apr 2, 2019 by bixb0012

Like • Show 0 Likes0
Comment • 18

I have a script that copies thousands of features, performs analysis and then deletes the copies. When I run this script in the Python window in ArcGIS Pro 2.3.1, each feature is copied and displayed in the TOC (ex. feature_new), and then is deleted. However, each time this happens, another copy with the original feature name and a number (ex. feature_old26) is created in the TOC, and remains after the script moves on to the next feature. So I have thousands of these in the TOC (see photo). If I use arcpy.env.addOutputsToMap = False in the script, the script immediately fails on copy features. ERROR 000732: Input Features: Dataset feature_old26 does not exist or is not supported. I know that this particular bug was logged almost a year ago (BUG-000116163), but I can't find any information about it.

The issue I have with this is that it's dramatically slowing down the Python script execution over time as hundreds or thousands of features build up in the TOC. I would run it in standalone IDLE, but for some reason it's faster in the Python window at first.

No one else has this question

Outcomes

18 Replies

dflater-esristaff Mar 28, 2019 11:11 AM
Mark Correct
Correct Answer
Hi,
The bug you linked, BUG-000116163, is not fixed yet. It deals specifically with arcpy.MakeFeatureLayer_management() when arcpy.env.addOutputToMap = False.
If your script is only doing a CopyFeatures then it might be a different issue that the script is failing. You do say after the copy the script "performs analysis" so I guess this may include arcpy.MakeFeatureLayer_management? You might be able to rewrite some steps in your script, as Make Feature Layer is no longer required as a pre-requisite for many tools like Add Join, Select Layer By Attribute, etc, as of Pro 2.2. https://pro.arcgis.com/en/pro-app/get-started/what-s-new-in-arcgis-pro-2-2.htm#ESRI_SECTION1_68F7BAF048AE4EF096C802EB7FD68817 (see the second bullet point)
3 people found this helpful
Like • Show 0 Likes0
Actions
MBDriscoll Mar 28, 2019 11:18 AM
Mark Correct
Correct Answer
You also might try creating the features in memory, there would be no need to delete them.
Like • Show 0 Likes0
Actions
LeviCecil Mar 28, 2019 12:35 PM
Mark Correct
Correct Answer
No, MakeFeatureLayer is not part of the script. It's peeling off and copying individual rooms from buildings, generating a tessellation for the room, select by location to count the hexagons and then moving on to the next room. All of those interim steps are deleted each time. The problem is, every time there's a copy of a room, another copy of the main feature class (all the buildings) is created in the TOC with a number at the end. That's filling up the TOC, and as far as I can tell, slowing the script down. When it starts, it's taking 40-45 seconds per room. After several hours it's up to around 3 minutes. Has anyone seen that happen before? It's very strange. Would creating the copies in memory solve this issue?
Like • Show 0 Likes0
Actions
bixb0012 Mar 28, 2019 12:36 PM
Mark Correct
Correct Answer
Please provide the code that is removing the layers from the TOC, or is the user doing it manually in the GUI?
Like • Show 0 Likes0
Actions
- LeviCecil @ bixb0012 on Mar 28, 2019 1:50 PM
  Mark Correct
  Correct Answer
  It's not removing them from the TOC, but the scratch.gdb. Pro is adding these copied layers to the TOC during the analysis, and then they are removed once they're deleted from the geodatabase. I'd rather they aren't added to the map at all, but due to that bug, I can't control it.
  
  To reiterate: I have one feature class with nearly 100 buildings that's been converted from CAD. The script goes through each building, copies out each room, runs analysis, and moves on to the next room. What's building up in the TOC is unexplained copies of the original feature class (see photo above) that are happening in tandem with the room feature copy. That is a feature class with 12400+ rows copied over and over again in the TOC. I believe it's clogging up the memory and slowing the Python script way down.
  
  Update: I believe this is a problem with the Project tool. I just tried to write the interim steps to memory and got this ERROR 000944: Output feature class cannot be in the in_memory workspace. Failed to execute (Project).
  
  Here is a better illustration of what's happening. The bottom feature is the original, the top feature is a single room that's been copied (based on an attribute selection from the main feature) and projected to the coordinate system based on the central meridian of the room. The middle feature is the copy of the original that remains in the TOC. Its source is identical to the main feature class, so I believe it's just being copied in memory by Pro.
  Like • Show 0 Likes0
  Actions
  - MBDriscoll @ LeviCecil on Apr 1, 2019 10:13 AM
    Mark Correct
    Correct Answer
    I believe you still need to make a feature layer from your feature class if you are going to run it in_memory.
    Like • Show 0 Likes0
    Actions
    - LeviCecil @ MBDriscoll on Apr 1, 2019 10:12 AM
      Mark Correct
      Correct Answer
      OK, but I still think there there is a bug here. There should not be extra copies of the original feature class clogging up the TOC.
      Like • Show 0 Likes0
      Actions
      - MBDriscoll @ LeviCecil on Apr 1, 2019 11:06 AM
        Mark Correct
        Correct Answer
        Hard to say without seeing the code. You may not be finishing the loop cleanly before moving onto the next one. If it is clean I recently learned that MapView.Redraw was introduced in 2.2, you might give that a try before moving onto the next one.
        Like • Show 0 Likes0
        Actions
  - bixb0012 @ LeviCecil on Apr 1, 2019 2:28 PM
    Mark Correct
    Correct Answer
    How or where are you executing this script? From the behavior you are seeing, I am guessing from the interactive Python window and not a script tool. Also, I agree with Matthew Driscoll, we really need to see some actual code to give more substantive feedback.
    1 person found this helpful
    Like • Show 0 Likes0
    Actions
    - LeviCecil @ bixb0012 on Apr 2, 2019 10:12 AM
      Mark Correct
      Correct Answer
      Yes, I was executing it in the Python window because it was faster, for a while. I just went back to running it in IDLE.
      
      Here is a truncated version of the script:
      
      import arcpy import os, sys import time from datetime import date, timedelta output_gdb = r'M:\\PROJECTS\\Projects_2018-19\\Map Requests\\Room Capacity Analysis\\room_capacity.gdb\\' scratch_gdb = r'M:\\PROJECTS\\Projects_2018-19\\Map Requests\\Room Capacity Analysis\\scratch.gdb\\' memory_hole = r'in_memory\\' arcpy.env.workspace = scratch_gdb arcpy.env.overwriteOutput = True arcpy.env.outputZFlag = "Disabled" #arcpy.env.addOutputsToMap = False fc = output_gdb + r'Rooms_w_SpaceID_AllSites_Proj' site = "" room = "" area_per = "" exclude_list = ["BUILDING_A","BUILDING_B"] sql_string = "WHERE UsableSF > 300 " print("Starting at " + str(datetime.datetime.now())) with arcpy.da.SearchCursor(fc, '*' ,sql_clause=(None, sql_string)) as cursor: for row in cursor: OID_field = arcpy.Describe(fc).OIDFieldName site = row[8] room_number = row[9] usable_square_footage = row[7] space_class = row[11] spatial_ref = row[21] if site in exclude_list: continue else: start_time = time.time() site_no_space = site.replace(' ','_') room_no_space = room_number.replace(' ','_') site_no_dash = site_no_space.replace('-','_') site_no_dot = site_no_dash.replace('.','_') room_no_dash = room_no_space.replace('-','') room_no_dot = room_no_dash.replace('.','') room_feature = memory_hole + site_no_dot.lower() + '_room_' + room_no_dot room_feature_proj = site_no_dot.lower() + "_room_" + room_no_dot + "_proj" where = '{0} = {1}'.format(arcpy.AddFieldDelimiters(fc,OID_field),row[0]) selection = arcpy.SelectLayerByAttribute_management(fc,"NEW_SELECTION",where) # select the current room feature arcpy.CopyFeatures_management(selection,room_feature) # output room to new feature class arcpy.Project_management(room_feature, scratch_gdb + room_feature_proj, spatial_ref) ## get room extent for grid description = arcpy.Describe(room_feature_proj) extent = description.extent thex_grid = memory_hole + room_feature_proj + "_thex_grid" area_per = "30 SquareFeet" footage = int(area_per[:3]) ## generate transverse hexagon grid polygon arcpy.management.GenerateTessellation(thex_grid, extent, "TRANSVERSE_HEXAGON", area_per, spatial_ref) ## Search for grid squares that fall completely within the room new_selection = arcpy.management.SelectLayerByLocation(thex_grid, "COMPLETELY_WITHIN", room_feature_proj, None, "NEW_SELECTION", "NOT_INVERT") count = arcpy.GetCount_management(new_selection) arcpy.AddField_management(room_feature_proj, "SQFT", "SHORT", 3, "", "", "SQFTPerStudent", "NULLABLE", "REQUIRED") arcpy.CalculateField_management(room_feature_proj, "SQFT", footage, "PYTHON_9.3") arcpy.AddField_management(room_feature_proj, "Count", "SHORT", 3, "", "", "OperationalCapacity", "NULLABLE", "REQUIRED") arcpy.CalculateField_management(room_feature_proj, "Count", count, "PYTHON_9.3") elapsed_time_secs = time.time() - start_time msg = "Execution time: %s" % timedelta(seconds=round(elapsed_time_secs)) print(msg) arcpy.Append_management(room_feature_proj, output_gdb + "\\rooms_with_operational_capacity","", "","","") del cursor
      Like • Show 0 Likes0
      Actions
      - bixb0012 @ LeviCecil on Apr 2, 2019 10:41 AM
        Mark Correct
        Correct Answer
        Does the script run as expected from IDLE? I assume so since your issue seems tied to TOC, but I hate assuming.
        Like • Show 0 Likes0
        Actions
        LeviCecil @ bixb0012 on Apr 2, 2019 10:58 AM
        Mark Correct
        Correct Answer
        It does run fine in IDLE, yes.
        Like • Show 0 Likes0
        Actions
        bixb0012 @ LeviCecil on Apr 2, 2019 1:46 PM
        Mark Correct
        Correct Answer
        What you are experiencing is a result of using the legacy "in_memory" space versus the new "memory" workspace in Pro: Write geoprocessing output to memory—ArcGIS Pro | ArcGIS Desktop
        in_memory datasets cannot be displayed on a map. If you run a geoprocessing tool from the Geoprocessing pane or Python window and write the output dataset to in_memory, after processing, the output dataset will be copied to the project geodatabase and that geodatabase dataset will be added to the map. Writing to the project geodatabase does not occur when in_memory datasets are intermediate and are not added to a map.
        
        I suggest you try using the new memory workspace.
        Like • Show 0 Likes0
        Actions
      - MBDriscoll @ LeviCecil on Apr 2, 2019 12:08 PM
        Mark Correct
        Correct Answer
        Nothing really sticking out here for the reason of the behavior. I really do think it is something in your code and not a bug for the following reason:
        
        The original example screen shot shows that a copy is getting created a lot but only every other third ends up in the table of contents. So it seems like it still creates a copy when it is in your exclude_list (continue) but does not go into your TOC.
        
        I do agree that it creating a new copy of the original on every loop and some of them getting added to the TOC is a major factor to slowing it down.
        Like • Show 0 Likes0
        Actions
        LeviCecil @ MBDriscoll on Apr 2, 2019 12:33 PM
        Mark Correct
        Correct Answer
        That's not what is happening at all. The exclude list is buildings (sites) from within the single feature class (Rooms_w_Space_ID_AllSites_Proj), which is nearly 100 buildings. All this list does is specify which buildings to skip over, it has nothing to do with the room copies. I just used generic names in the example, but there are over a dozen buildings excluded. Nowhere in the script do I specify a copy of the main feature class. This is happening inside Pro in the TOC when this script is run from the Python window.
        
        I'll explain again: each room from each building is selected from the main feature class (Rooms_w_Space_ID_AllSites_Proj). This selection is copied as a feature, projected, analyzed and then all these steps are deleted. Next room, repeat. What's building up in the TOC are numbered copies of the main feature class (which shouldn't be happening at all). It's happening every single time that a room from the feature class is copied and projected to the central meridian of the room. There is something going on with the project tool in Pro. The photo above just shows the results of one iteration of this. There are thousands of these copies building up in the TOC as the script runs (see photo from my initial post).
        Like • Show 0 Likes0
        Actions
        MBDriscoll @ LeviCecil on Apr 2, 2019 1:26 PM
        Mark Correct
        Correct Answer
        What I was pointing out was that if you look at the numbers, on the original photo, they are not +1, this shows that the copies of the main feature classes are getting copied more than are getting put on the TOC. I simplified the code and ran it and was not able to reproduce this problem.
        
        Also, after reading the Project documentation I found I steered you down a wrong path, you cannot use the in_memory for that particular tool. Which is why you got that error.
        
        import arcpy arcpy.env.overwriteOutput = True output_gdb = r"C:\\Users\\mdriscoll\\Desktop\\New Folder\\abc\\Default.gdb\\" exclude_list = ["A","E"] fc = output_gdb + "pnt" with arcpy.da.SearchCursor(fc,"*") as cursor:     copy = output_gdb + "copy"     outCS = arcpy.SpatialReference('NAD 1983 UTM Zone 11N')     outputCS = output_gdb + "copy_proj"     for row in cursor:         field = row[2]         if field in exclude_list:             continue         else:             arcpy.CopyFeatures_management(fc,copy)             arcpy.Project_management(copy, outputCS, outCS)             print(field)
        
        Like • Show 0 Likes0
        Actions
        LeviCecil @ MBDriscoll on Apr 2, 2019 1:31 PM
        Mark Correct
        Correct Answer
        OK, but how can I simplify my script to avoid this? I see you're not selecting a layer by attribute before your copy and project steps.
        Like • Show 0 Likes0
        Actions
        MBDriscoll @ LeviCecil on Apr 2, 2019 1:42 PM
        Mark Correct
        Correct Answer
        I was just testing the Copy and Project to show that you are perhaps going down the wrong path. I did not see anything that stuck out. If I were you I might comment out as much as you can and maybe test it line by line until the behavior is reproduced to find your smoking gun.
        Like • Show 0 Likes0
        Actions

GeoNet

The Esri Community

Copy and Project Features Slowing Down Script in Python Window

Outcomes

Related Content

Recommended Content