Hello, I have the following script. It works as expected, but it takes around 1.5 seconds to run each time. Doesn't seem to matter if it's running against 20 parcels or 5000. Are there any suggestions as to how I can improve the performance of this script. Thanks import arcpy import numpy as np try : arcpy . env . overwriteOutput = True #Create in memory tables: arcpy . CreateTable_management ( "in_memory" , "tableSelRecs" ) arcpy . AddField_management ( r 'in_memory\tableSelRecs' , "USE_CATEGORY" , "TEXT" , field_length = 35 ) arcpy . AddField_management ( r 'in_memory\tableSelRecs' , "SHAPE_Area" , "Double" ) arcpy . CreateTable_management ( "in_memory" , "tableSumRecs" ) arcpy . AddField_management ( r 'in_memory\tableSumRecs' , "USE_CATEGORY" , "TEXT" , field_length = 35 ) arcpy . AddField_management ( r 'in_memory\tableSumRecs' , "SHAPE_Area" , "Double" ) mxd = arcpy . mapping . MapDocument ( 'CURRENT' ) df = mxd . activeDataFrame totalArea = 0 totalResArea = 0 totalParcels = 0 totalResParcels = 0 lu_layer = arcpy . mapping . ListLayers ( mxd , "LandUse_UseCategory" , df ) [ 0 ] #use da.SearchCursor to access necessary fields from selected records with arcpy . da . SearchCursor ( lu_layer , [ 'USE_CATEGORY' , 'SHAPE@AREA' ] ) as cursor : #Setup insert cursor to insert selected polygons into tableSelRecs insCursor = arcpy . da . InsertCursor ( r 'in_memory\tableSelRecs' , [ 'USE_CATEGORY' , 'SHAPE_Area' ] ) #Isert selected rows into insCursor; #Get total area of selected records; #Get a count of total parcels for row in cursor : if row [ 0 ] != 'Invalid' : insCursor . insertRow ( ( row [ 0 ] , row [ 1 ] ) ) totalArea += row [ 1 ] totalParcels += 1 #If the block is of a 'Residential' USE_CATEGORY type then add that value into the Residential DI calculation variable if row [ 0 ] in ( 'Residential - Low Density' , 'Residential - Medium Density' , 'Residential - High Density' ) : totalResArea += row [ 1 ] totalResParcels += 1 #use summarystatistics against tableSelRecs to get totals per USE_CATEGORY and write results to tableSumRecs arcpy . Statistics_analysis ( r 'in_memory\tableSelRecs' , r 'in_memory\tableSumRecs' , [ [ "SHAPE_Area" , "SUM" ] ] , "USE_CATEGORY" ) with arcpy . da . SearchCursor ( r 'in_memory\tableSumRecs' , [ 'USE_CATEGORY' , 'SUM_SHAPE_Area' ] ) as selCur : #list to store calculation for each USE_CATEGORY value used in DI calculation interimValue = [ 0 ] #list to store calculation for each USE_CATEGORY value used in RDI calculation interimResValue = [ 0 ] for sRow in selCur : arcpy . AddMessage ( 'USE_CATEGORY: {}; AREA: {}' . format ( sRow [ 0 ] , sRow [ 1 ] ) ) #divide the USE_CATEGORY area by the total community area then square that number & append it to the list interimValue . append ( np . square ( sRow [ 1 ] / totalArea ) ) if sRow [ 0 ] in ( 'Residential - Low Density' , 'Residential - Medium Density' , 'Residential - High Density' ) : interimResValue . append ( np . square ( sRow [ 1 ] / totalResArea ) ) #sum the values in the list then subtract that value from 1 to get the DI for the selected blocks DI = 1 - sum ( interimValue ) #sum the values in the list then subtract that value from 1 to get the RDI for the selected blocks RDI = 1 - sum ( interimResValue ) arcpy . AddMessage ( 'Diversity Index: ' + str ( DI ) ) arcpy . AddMessage ( 'Residential Diversity Index: ' + str ( RDI ) ) arcpy . AddMessage ( 'Total Blocks Selected: ' + str ( totalParcels ) ) arcpy . AddMessage ( 'Total Blocks Area: ' + str ( totalArea ) ) arcpy . AddMessage ( 'Total Res Blocks Selected: ' + str ( totalResParcels ) ) arcpy . AddMessage ( 'Total Res Blocks Area: ' + str ( totalResArea ) ) arcpy . AddMessage ( 'NOTE: Any blocks with a use category of Invalid are not included in any calculations.' ) #Delete in memory tables arcpy . Delete_management ( r 'in_memory\tableSelRecs' ) arcpy . Delete_management ( r 'in_memory\tableSumRecs' ) except arcpy . ExecuteError : print ( arcpy . GetMessages ( 2 ) )

Lots of 'speed' issues this week, so * software, version * machine type and operating system * ram, if you are using Pro or 64 bit geoprocessing in arcmap * is other software running? * are you monitoring resources? * is everything indexed? * same performance in_memory vs to disk? * had you considered collecting in the searchcursor, then processing in a separate insertcursor ?(would take some jigging), this seems to be a not-recommended by some Suggestions * break everything into functions (def) * you can use a 'decorator' or profiler on a def to see where the bottleneck is * you don't need numpy. * throw timing steps and print statements if you don't want to use a decorator * dump the try-except, there is no need since the errors that are returned are readily identifiable unless they are obscure more questions, but this will be a start.

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Python Script & Performance Speed

cwh

Hello,

I have the following script. It works as expected, but it takes around 1.5 seconds to run each time. Doesn't seem to matter if it's running against 20 parcels or 5000. Are there any suggestions as to how I can improve the performance of this script.

Thanks

<SPAN class="keyword token">import</SPAN> arcpy
<SPAN class="keyword token">import</SPAN> numpy <SPAN class="keyword token">as</SPAN> np

<SPAN class="keyword token">try</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>env<SPAN class="punctuation token">.</SPAN>overwriteOutput <SPAN class="operator token">=</SPAN> <SPAN class="token boolean">True</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#Create in memory tables:</SPAN>
&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>CreateTable_management<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"in_memory"</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"tableSelRecs"</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddField_management<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'in_memory\tableSelRecs'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"USE_CATEGORY"</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"TEXT"</SPAN><SPAN class="punctuation token">,</SPAN> field_length<SPAN class="operator token">=</SPAN><SPAN class="number token">35</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddField_management<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'in_memory\tableSelRecs'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"SHAPE_Area"</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"Double"</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>CreateTable_management<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"in_memory"</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"tableSumRecs"</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddField_management<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'in_memory\tableSumRecs'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"USE_CATEGORY"</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"TEXT"</SPAN><SPAN class="punctuation token">,</SPAN> field_length<SPAN class="operator token">=</SPAN><SPAN class="number token">35</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddField_management<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'in_memory\tableSumRecs'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"SHAPE_Area"</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"Double"</SPAN><SPAN class="punctuation token">)</SPAN>

&nbsp;&nbsp;&nbsp; mxd <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>mapping<SPAN class="punctuation token">.</SPAN>MapDocument<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'CURRENT'</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; df <SPAN class="operator token">=</SPAN> mxd<SPAN class="punctuation token">.</SPAN>activeDataFrame
&nbsp;&nbsp;&nbsp; totalArea <SPAN class="operator token">=</SPAN> <SPAN class="number token">0</SPAN>
&nbsp;&nbsp;&nbsp; totalResArea <SPAN class="operator token">=</SPAN> <SPAN class="number token">0</SPAN>
&nbsp;&nbsp;&nbsp; totalParcels <SPAN class="operator token">=</SPAN> <SPAN class="number token">0</SPAN>
&nbsp;&nbsp;&nbsp; totalResParcels <SPAN class="operator token">=</SPAN> <SPAN class="number token">0</SPAN>

&nbsp;&nbsp;&nbsp; lu_layer <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>mapping<SPAN class="punctuation token">.</SPAN>ListLayers<SPAN class="punctuation token">(</SPAN>mxd<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"LandUse_UseCategory"</SPAN><SPAN class="punctuation token">,</SPAN> df<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#use da.SearchCursor to access necessary fields from selected records</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>lu_layer<SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'USE_CATEGORY'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'SHAPE@AREA'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#Setup insert cursor to insert selected polygons into tableSelRecs</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; insCursor <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>InsertCursor<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'in_memory\tableSelRecs'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'USE_CATEGORY'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'SHAPE_Area'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#Isert selected rows into insCursor;</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#Get total area of selected records;</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#Get a count of total parcels</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">!=</SPAN> <SPAN class="string token">'Invalid'</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; insCursor<SPAN class="punctuation token">.</SPAN>insertRow<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">(</SPAN>row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN>row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; totalArea <SPAN class="operator token">+=</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; totalParcels <SPAN class="operator token">+=</SPAN> <SPAN class="number token">1</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#If the block is of a 'Residential' USE_CATEGORY type then add that value into the Residential DI calculation variable</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="keyword token">in</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Residential - Low Density'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'Residential - Medium Density'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'Residential - High Density'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; totalResArea <SPAN class="operator token">+=</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; totalResParcels <SPAN class="operator token">+=</SPAN> <SPAN class="number token">1</SPAN>

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#use summarystatistics against tableSelRecs to get totals per USE_CATEGORY and write results to tableSumRecs</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>Statistics_analysis<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'in_memory\tableSelRecs'</SPAN><SPAN class="punctuation token">,</SPAN> r<SPAN class="string token">'in_memory\tableSumRecs'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">"SHAPE_Area"</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"SUM"</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"USE_CATEGORY"</SPAN><SPAN class="punctuation token">)</SPAN>

&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'in_memory\tableSumRecs'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'USE_CATEGORY'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'SUM_SHAPE_Area'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> selCur<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#list to store calculation for each USE_CATEGORY value used in DI calculation</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; interimValue <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#list to store calculation for each USE_CATEGORY value used in RDI calculation</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; interimResValue <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">for</SPAN> sRow <SPAN class="keyword token">in</SPAN> selCur<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'USE_CATEGORY: {}; AREA: {}'</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>sRow<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN>sRow<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#divide the USE_CATEGORY area by the total community area then square that number & append it to the list</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; interimValue<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>np<SPAN class="punctuation token">.</SPAN>square<SPAN class="punctuation token">(</SPAN>sRow<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="operator token">/</SPAN>totalArea<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN> 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> sRow<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="keyword token">in</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Residential - Low Density'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'Residential - Medium Density'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'Residential - High Density'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; interimResValue<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>np<SPAN class="punctuation token">.</SPAN>square<SPAN class="punctuation token">(</SPAN>sRow<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="operator token">/</SPAN>totalResArea<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#sum the values in the list then subtract that value from 1 to get the DI for the selected blocks</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DI <SPAN class="operator token">=</SPAN> <SPAN class="number token">1</SPAN> <SPAN class="operator token">-</SPAN> sum<SPAN class="punctuation token">(</SPAN>interimValue<SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#sum the values in the list then subtract that value from 1 to get the RDI for the selected blocks</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RDI <SPAN class="operator token">=</SPAN> <SPAN class="number token">1</SPAN> <SPAN class="operator token">-</SPAN> sum<SPAN class="punctuation token">(</SPAN>interimResValue<SPAN class="punctuation token">)</SPAN> 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Diversity Index: '</SPAN> <SPAN class="operator token">+</SPAN> str<SPAN class="punctuation token">(</SPAN>DI<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Residential Diversity Index: '</SPAN> <SPAN class="operator token">+</SPAN> str<SPAN class="punctuation token">(</SPAN>RDI<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Total Blocks Selected: '</SPAN> <SPAN class="operator token">+</SPAN> str<SPAN class="punctuation token">(</SPAN>totalParcels<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Total Blocks Area: '</SPAN> <SPAN class="operator token">+</SPAN> str<SPAN class="punctuation token">(</SPAN>totalArea<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Total Res Blocks Selected: '</SPAN> <SPAN class="operator token">+</SPAN> str<SPAN class="punctuation token">(</SPAN>totalResParcels<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Total Res Blocks Area: '</SPAN> <SPAN class="operator token">+</SPAN> str<SPAN class="punctuation token">(</SPAN>totalResArea<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'NOTE: Any blocks with a use category of Invalid are not included in any calculations.'</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#Delete in memory tables</SPAN>
&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>Delete_management<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'in_memory\tableSelRecs'</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; arcpy<SPAN class="punctuation token">.</SPAN>Delete_management<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'in_memory\tableSumRecs'</SPAN><SPAN class="punctuation token">)</SPAN>

<SPAN class="keyword token">except</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>ExecuteError<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>arcpy<SPAN class="punctuation token">.</SPAN>GetMessages<SPAN class="punctuation token">(</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Find more posts tagged with

Accepted answers

DanPatterson_Retired

Lots of 'speed' issues this week, so

software, version
machine type and operating system
ram, if you are using Pro or 64 bit geoprocessing in arcmap
is other software running?
are you monitoring resources?
is everything indexed?
same performance in_memory vs to disk?
had you considered collecting in the searchcursor, then processing in a separate insertcursor ?(would take some jigging), this seems to be a not-recommended by some

Suggestions

break everything into functions (def)
you can use a 'decorator' or profiler on a def to see where the bottleneck is
you don't need numpy.
throw timing steps and print statements if you don't want to use a decorator
dump the try-except, there is no need since the errors that are returned are readily identifiable unless they are obscure

more questions, but this will be a start.

All comments

deleted-user-gdY-GmN0uDXr

I have found (from reading GeoNet posts and in my own experience) that from arcpy import * is quicker than import arcpy.

cwh

Hi Blake, yes I found that removing the reference to the mxd file and instead using parameters to pass in the feature layer improved overall performance down to around 0.7 seconds. I think the calculations perform very fast it’s just the overhead getting there.

blake.terhune

Did any of those suggestions in particular make a significant difference in the performance of your script?

cwh

Thanks for the tip Soren. Ya, I’m happy with the performance now. Moved on with getting it working in the Web AppBuilder environment.

SørenNielsen

Hi Chris,

I have had a lot of success using arcpy.SetLogHistory(False) when updating feature classes in the SDE. It disables the logging of the geoprocessing tools. I think you can improve the speed of your script a little. But 1.5 seconds is very fast!

Best regards

Søren

DanPatterson_Retired

actually np.sqrt takes no more time than math.sqrt ... both measure in micro seconds.

If you want to speed up numeric calculations on fields for large tables (ie many records) pull the field out, use numpy to vectorize the calculation then use arcpy.da.extendtable to add the result back..

cwh

When you say it takes 1.5 seconds each time, is that the time when freshly initialized and ran, or does it take that long even if run successively in the same process/session?

It does run a little faster on subsequent runs.

Depending on which ArcPy components are being used, initializing/loading all of them into a clean Python process/session make take a second. You might be seeing the same times between 20 and 5000 parcels because the tool runs so quick that the time you are seeing is the overhead of initializing objects and libraries.

I think you are correct.

If you are going to time your functions, it might be worth timing how long it takes for your import statements as well.

Good point, would be interesting to see.

Thanks Joshua

ChrisSnyder

Well that's good news... Yeah, I'm finding that some of the arcpy.mapping functions aren't the fastest horses on the track. Onwards and upwards!

cwh

Why is there a need to write data to the in_memory tables? Seems like the stats you are calculating only rely on read data. Perhaps a dictionary could serve the purpose of you intermediate in_memory tables?

I believe when I first started working on this script (and created a different geonet thread) the suggestions were to use intermediate tables with summary statistics or use a dictionary. I got the script working with summary statistics (with a plan later to try to get it working with a dictionary but as yet haven't got back to it).

Significant bottlenecks I see are:

1. Creating the intermediate tables, adding fields to them, and inserting values

Agreed.

2. Writing data to numpy arrays from the cursors. Seems like the np.square() calculations are entirely doable outside of numpy. Certainly more convenient in np though.

Agree on both points.

3. Using arcpy.mapping to get a hook into the layer. Maybe you could just have a toolbox w/ an input feature layer variable that had a default value set to your desired layer name?

This is a change that I have made as I had the python toolbox published as a gp service to use as a geoprocessing widget in web appbuilder and without inputs and outputs it's basically useless. So making this change of using inputs/outputs and removing the need to hook into the layer has brought the script run time down to 0.3 - 0.4 seconds.

Thanks for your input Chris

JoshuaBixby

When you say it takes 1.5 seconds each time, is that the time when freshly initialized and ran, or does it take that long even if run successively in the same process/session?

If you are going to time your functions, it might be worth timing how long it takes for your import statements as well.

ChrisSnyder

Significant bottlenecks I see are:

1. Creating the intermediate tables, adding fields to them, and inserting values

2. Writing data to numpy arrays from the cursors. Seems like the np.square() calculations are entirely doable outside of numpy. Certainly more convenient in np though.

3. Using arcpy.mapping to get a hook into the layer. Maybe you could just have a toolbox w/ an input feature layer variable that had a default value set to your desired layer name?

cwh

So I talked to our server guys as the plan is to have this script published as a gp service which I can then use in web appbuilder. They told me that 1.5 seconds is very good. So maybe I should move forward with having the script published. I'm thankful for the comments here as I have taken away some good learnings and some pointers to help me improve the script. thanks

cwh

Ok, thanks Dan. I see how that could help me identify any bottleneck.

DanPatterson_Retired

Chris... for decorators, you need to have functions... here is an example script setup, that basically returns a list of numbers. I put sleepy time in there to get a meaningful time back. The if __name__ == '__main__' section calls and runs the script. if your ide uses IPython (or others) you can use things like %timeit to run lines of code or functions etc. Decorators are selective in that you can choose to 'decorate' just the functions that you want (line 20 is the 'decorator' kind of like a special line above a function)

<SPAN class="keyword token">def</SPAN> <SPAN class="token function">time_deco</SPAN><SPAN class="punctuation token">(</SPAN>func<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>&nbsp; <SPAN class="comment token"># timing originally</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="string token">"""timing decorator function
&nbsp;&nbsp;&nbsp; :print("\n&nbsp; print results inside wrapper or use <return> ... ")
&nbsp;&nbsp;&nbsp; """</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">import</SPAN> time
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">from</SPAN> functools <SPAN class="keyword token">import</SPAN> wraps

&nbsp;&nbsp;&nbsp; @wraps<SPAN class="punctuation token">(</SPAN>func<SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">def</SPAN> <SPAN class="token function">wrapper</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="operator token">*</SPAN>args<SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">**</SPAN>kwargs<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; t0 <SPAN class="operator token">=</SPAN> time<SPAN class="punctuation token">.</SPAN>perf_counter<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># start time</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; result <SPAN class="operator token">=</SPAN> func<SPAN class="punctuation token">(</SPAN><SPAN class="operator token">*</SPAN>args<SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">**</SPAN>kwargs<SPAN class="punctuation token">)</SPAN>&nbsp; <SPAN class="comment token"># ... run the function ...</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; t1 <SPAN class="operator token">=</SPAN> time<SPAN class="punctuation token">.</SPAN>perf_counter<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># end time</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; dt <SPAN class="operator token">=</SPAN> t1 <SPAN class="operator token">-</SPAN> t0
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">"\nTiming function for... {}"</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>func<SPAN class="punctuation token">.</SPAN>__name__<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">"&nbsp; Time: {: <8.2e}s for {:,} objects"</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>dt<SPAN class="punctuation token">,</SPAN> len<SPAN class="punctuation token">(</SPAN>result<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> result&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># return the result of the function</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> dt&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># return delta time</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> wrapper

@time_deco
<SPAN class="keyword token">def</SPAN> <SPAN class="token function">_demo</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="string token">"""
&nbsp;&nbsp;&nbsp; : -
&nbsp;&nbsp;&nbsp; """</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">import</SPAN> time
&nbsp;&nbsp;&nbsp; time<SPAN class="punctuation token">.</SPAN>sleep<SPAN class="punctuation token">(</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; result <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">2</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3</SPAN><SPAN class="punctuation token">]</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> result

<SPAN class="comment token"># ----------------------------------------------------------------------</SPAN>
<SPAN class="comment token"># __main__ .... code section</SPAN>
<SPAN class="keyword token">if</SPAN> __name__ <SPAN class="operator token">==</SPAN> <SPAN class="string token">"__main__"</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="string token">"""Optionally...
&nbsp;&nbsp;&nbsp; : - print the script source name.
&nbsp;&nbsp;&nbsp; : - run the _demo
&nbsp;&nbsp;&nbsp; """</SPAN>
<SPAN class="comment token">#&nbsp;&nbsp;&nbsp; print("Script... {}".format(script))</SPAN>
&nbsp;&nbsp;&nbsp; result <SPAN class="operator token">=</SPAN> _demo<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Now if you don't want a full-fledged timer, examine lines 10-13, you can start, stop and get time differences and print the result out.. just like the decorator. perfcounter is the preferred counter for windows.

cwh

I have no idea what a decorator is.

cwh

A good point. It just seems odd to me that it's no faster when run with a small number of parcels.

cwh

Thanks Dan for the comments and suggestions.

DanPatterson_Retired

Good point...

What IDE are you using?

Some have the option to do 'module reloading' everytime a script is run. Check your ide to see.

That will affect timing a whole load, and you will need a decorator to time function (I can provide if you do a lot of testing)

jskinner

but it takes around 1.5 seconds to run each time

Is that correct? Are you looking to have it run even faster than 1.5 seconds?

DanPatterson_Retired

Lots of 'speed' issues this week, so

software, version
machine type and operating system
ram, if you are using Pro or 64 bit geoprocessing in arcmap
is other software running?
are you monitoring resources?
is everything indexed?
same performance in_memory vs to disk?
had you considered collecting in the searchcursor, then processing in a separate insertcursor ?(would take some jigging), this seems to be a not-recommended by some

Suggestions

break everything into functions (def)
you can use a 'decorator' or profiler on a def to see where the bottleneck is
you don't need numpy.
throw timing steps and print statements if you don't want to use a decorator
dump the try-except, there is no need since the errors that are returned are readily identifiable unless they are obscure

more questions, but this will be a start.

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.