nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Why would my additional MakeTableView_management() calls take drastically longer?

Karyn_Kerdolff

I'm working on a Python script that involves selecting parcels from a feature class and I've run into some odd behavior in my sanity checks. In the code below I loop through the list of specified tax ID numbers and use the MakeTableView trick to get a count of elements matching a specific "where" clause. If I only give it one tax ID, the script finishes in a couple of seconds. If I give it two, it gets stuck on MakeTableView for about 4 minutes, and this seems to increase by about 4 minutes for each additional tax ID. I've tried explicitly deleting the resulting table view as soon as I'm finished with it, but that made no difference.

Ultimately, I changed the script to use a search cursor and the "where" clause to give me the same functionality, but this is just bouncing around the back of my head—why would the subsequent MakeTableView calls perform so much slower than the first?

Edit for clarity: TID is a list of tax ID's obtained by splitting a multi-value GetParameterAsText().

<SPAN class="comment token"># Add all desired parcels to selection</SPAN>
<SPAN class="keyword token">for</SPAN> tid <SPAN class="keyword token">in</SPAN> TID<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> tid <SPAN class="operator token">and</SPAN> tid <SPAN class="operator token">!=</SPAN> "<SPAN class="comment token">#":</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># Make sure the parcel ID is formatted correctly</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> <SPAN class="operator token">not</SPAN> re<SPAN class="punctuation token">.</SPAN>match<SPAN class="punctuation token">(</SPAN>pattern<SPAN class="punctuation token">,</SPAN> tid<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">raise</SPAN> GPException<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"Input Parcel IDs must be in the format "</SPAN> <SPAN class="operator token">+</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="string token">"YY-YYY-YYYY, where Y is a single digit number."</SPAN> <SPAN class="operator token">+</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="string token">" For example, 06-019-0009."</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># Make sure parcel ID is a valid parcel</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; where <SPAN class="operator token">=</SPAN> TID_field <SPAN class="operator token">+</SPAN> <SPAN class="string token">" = '"</SPAN> <SPAN class="operator token">+</SPAN> tid <SPAN class="operator token">+</SPAN> <SPAN class="string token">"'"</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>MakeTableView_management<SPAN class="punctuation token">(</SPAN>parcels<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"in_memory/temp"</SPAN><SPAN class="punctuation token">,</SPAN> where<SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>GetCount_management<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"in_memory/temp"</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="operator token"><</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">raise</SPAN> GPException<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"Cannot find parcel ID "</SPAN> <SPAN class="operator token">+</SPAN> tid <SPAN class="operator token">+</SPAN> <SPAN class="string token">" in parcel "</SPAN> <SPAN class="operator token">+</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="string token">"list."</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>Delete_management<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"in_memory/temp"</SPAN><SPAN class="punctuation token">)</SPAN>&nbsp;&nbsp;&nbsp; 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># Finally, add it to the selection if it's passed all the checks</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>SelectLayerByAttribute_management<SPAN class="punctuation token">(</SPAN>parcels<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"ADD_TO_SELECTION"</SPAN><SPAN class="punctuation token">,</SPAN> where<SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Find more posts tagged with

Accepted answers

All comments

blake.terhune

I think this is the direction I would go too. It seems unnecessary to have this in a loop. Here's my take.

TID <SPAN class="operator token">=</SPAN> tuple<SPAN class="punctuation token">(</SPAN>str<SPAN class="punctuation token">(</SPAN>i<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">for</SPAN> i <SPAN class="keyword token">in</SPAN> TID<SPAN class="punctuation token">)</SPAN>
where <SPAN class="operator token">=</SPAN> <SPAN class="string token">"{} in{}"</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>TID_field<SPAN class="punctuation token">,</SPAN> TID<SPAN class="punctuation token">)</SPAN>
arcpy<SPAN class="punctuation token">.</SPAN>MakeTableView_management<SPAN class="punctuation token">(</SPAN>parcels<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"parcels_tblvw"</SPAN><SPAN class="punctuation token">,</SPAN> where<SPAN class="punctuation token">)</SPAN>
arcpy<SPAN class="punctuation token">.</SPAN>SelectLayerByAttribute_management<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"parcels_tblvw"</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"NEW_SELECTION"</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Karyn_Kerdolff

Good to know, thank you!

JoshuaBixby

A comment on syntax. With Make Table View or any of the similar tools, the second field is the name of the layer to be created, just the name. In your code above, you are using "in_memory/temp", which appears to work, but I suggest against adding "in_memory/" in front of the name for 2 reasons: 1) it doesn't do anything, i.e., views are already created in memory and adding "in_memory/" doesn't really do anything besides make the name of the view longer, and 2) adding "in_memory/" implies a workspace is needed by the tool when it isn't, which creates readability and potential code maintenance issues.

Karyn_Kerdolff

That is definitely a more elegant solution than mine. I'm doing a check with an explicit error because eventually I want to expose my script to the public through a geoprocessing service and I want to give meaningful feedback on errors.

Karyn_Kerdolff

Thanks for the suggestions, everyone. It definitely looks like it doesn't like being in a loop. Extracting it to a function and calling it in a loop results in longer run times (in this particular case), so there's something going on there that seems to confirm the warnings about loops.

KimOllivier

I never use an arcpy tool in a loop. They are not designed to do that.

You could assemble your list of FIDS into a list, convert to a string and use a SQL "IN" statement to select in one step.

Faster, easier to understand, easier to debug and easier to create.

Why do you need to check if the parcel id exists? If it doesn't it won't matter in an IN list.

# list of valid ids (do a regular expression in a list comprehension to validate
print(len(FIDS)) # just to see how many, limit 50,000 should be fine, if more then partition, can use 1M if integers not strings
the_list = str(tuple(FIDS)).replace('u','') # strip off any unicode tags
sql = "{} in {}".format(the_field,the_list)
print (sql[0:40], sql[-40:]) # check syntax
# make sure FID is indexed for performance
arcpy.SelectLayerByAttribute_management(parcels, "ADD_TO_SELECTION", sql)

JoshuaBixby

I have run into issues with certain geoprocessing tools getting bogged down like what you describe here when looping. I think it is tied to some memory management or garbage collection issue at the interface between the tool and ArcPy. Try taking lines 11-13 and wrapping them in a stand-alone function, and then call that function from within your loop. See if that does away with the incremental time creep you are seeing.

Karyn_Kerdolff

My browser is showing line 14 as the continuation of the exception's message, so I assume we're talking about the Delete_management() line. I don't think it's indented wrong—I am trying to delete the table view regardless of the outcome of the "if GetCount..." as an attempt to solve the problem under the assumption that trying to create a table view with the same path as an existing table view could be causing issues. However, including or removing this line hasn't changed the behavior.

DanPatterson_Retired

so is line 14 indented wrong then? it seems you create the in-memory table, then try to delete the workspace there

Karyn_Kerdolff

Dan, TID is a list of input tax IDs that comes from a GetParamterAsText() call on a multi-value parameter. If I am not misunderstanding anything, this means there's only one loop and its number of executions is equal to the number of items in that list. I've confirmed this by debugging it and stepping through each line of code. Everything works as I'd expect, but the program stops at line 11 (MakeTableManagement) for several minutes the second (third, fourth, etc) time through the loop but not the first.

(edit: line number)

DanPatterson_Retired

everything has to cycle through your first loop, hence your incremental time creep as you add more to the quest. Perhaps your indentation is completely off but I see no reason to cycle through every feature when you really only want to cycle through the features that match your query

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.