Latest Contributions by ChrisHolmes

Thanks for the tip Soren. Ya, I’m happy with the performance now. Moved on with getting it working in the Web AppBuilder environment. ... View more

So thought I'd update where things are at with this. Getting closer but still not working. I changed the code as follows so that the toolbox has inputs and outputs (code at bottom). When I use the select widget to select blocks it seems as if the selection is not being used as input to the gp widget. As you can see from the gp results it is using 33291 blocks in the calculations (the layer has a total of 34646 blocks). So I'm not sure where the 33291 is coming from. Also I thought I would see "Set as input of geoprocessing" as a select option, but I only see the following: Any thoughts appreciated as to what I may be missing to get the gp widget to use the selected blocks. Hoping we're getting close. Thanks everyone! ############################################ ###Start of class for DiversityIndex toolbox ############################################ class Toolbox(object):     def __init__(self):         """Define the toolbox (the name of the toolbox is the name of the         .pyt file)."""         self.label = "DiversityIndex"         self.alias = ""         # List of tool classes associated with this toolbox         self.tools = [DiversityIndex] ############################################ ###Start of class for DiversityIndex tool ############################################ class DiversityIndex(object):     def __init__(self):         """Define the tool (tool name is the name of the class)."""         self.label = "Diversity Index"         self.description = "Calculate the land use diversity index based on selected blocks."         self.canRunInBackground = False     def getParameterInfo(self):         #Define parameter definitions         # First parameter         param0 = arcpy.Parameter(             displayName="Input Features",             name="in_features",             datatype="GPFeatureLayer",             parameterType="Required",             direction="Input")         param0.value = 'LandUse_UseCategory'         # Second parameter         param1 = arcpy.Parameter(             displayName="Diversity Index",             name="diversity_index",             datatype="GPDouble",             parameterType="Derived",             direction="Output")         # Third parameter         param2 = arcpy.Parameter(             displayName="Residential Diversity Index",             name="residential_diversity_index",             datatype="GPDouble",             parameterType="Derived",             direction="Output")         # Fourth parameter         param3 = arcpy.Parameter(             displayName="Total Blocks Selected",             name="total_blocks_selected",             datatype="GPLong",             parameterType="Derived",             direction="Output")         # Fifth parameter         param4 = arcpy.Parameter(             displayName="Total Blocks Area",             name="total_blocks_area",             datatype="GPDouble",             parameterType="Derived",             direction="Output")         # Sixth parameter         param5 = arcpy.Parameter(             displayName="Total Residential Blocks Selected",             name="total_residential_blocks_selected",             datatype="GPLong",             parameterType="Derived",             direction="Output")         # Seventh parameter         param6 = arcpy.Parameter(             displayName="Total Residential Blocks Area",             name="total_residential_blocks_area",             datatype="GPDouble",             parameterType="Derived",             direction="Output")         # Eighth parameter         param7 = arcpy.Parameter(             displayName="Note",             name="note",             datatype="GPString",             parameterType="Derived",             direction="Output")         param7.value = 'NOTE: Any blocks with a use category of Invalid are not included in any calculations.'         params = [param0, param1, param2, param3, param4, param5, param6, param7]         return params     def isLicensed(self):         """Set whether tool is licensed to execute."""         return True     def updateParameters(self, parameters):         """Modify the values and properties of parameters before internal         validation is performed.  This method is called whenever a parameter         has been changed."""         return     def updateMessages(self, parameters):         """Modify the messages created by internal validation for each tool         parameter.  This method is called after internal validation."""         return     def execute(self, parameters, messages):         """The source code of the tool."""         import numpy as np         try:           arcpy.env.overwriteOutput = True           inFeatures = parameters[0].valueAsText           #Create in memory tables:           arcpy.CreateTable_management("in_memory", "tableSelRecs")           arcpy.AddField_management(r'in_memory\tableSelRecs', "USE_CATEGORY", "TEXT", field_length=35)           arcpy.AddField_management(r'in_memory\tableSelRecs', "SHAPE_Area", "Double")           arcpy.CreateTable_management("in_memory", "tableSumRecs")           arcpy.AddField_management(r'in_memory\tableSumRecs', "USE_CATEGORY", "TEXT", field_length=35)           arcpy.AddField_management(r'in_memory\tableSumRecs', "SHAPE_Area", "Double")           totalArea = 0           totalResArea = 0           totalParcels = 0           totalResParcels = 0           #use da.SearchCursor to access necessary fields from selected records           with arcpy.da.SearchCursor(inFeatures,['USE_CATEGORY','SHAPE@AREA']) as cursor:               #Setup insert cursor to insert selected polygons into tableSelRecs               insCursor = arcpy.da.InsertCursor(r'in_memory\tableSelRecs', ['USE_CATEGORY','SHAPE_Area'])               #Isert selected rows into insCursor; Get total area of selected records; Get a count of total parcels               for row in cursor:                   if row[0] != 'Invalid':                       insCursor.insertRow((row[0],row[1]))                       totalArea += row[1]                       totalParcels += 1                   #If the block is of a 'Residential' USE_CATEGORY type then add that value into the Residential DI calculation variable                   if row[0] in('Residential - Low Density','Residential - Medium Density','Residential - High Density'):                       totalResArea += row[1]                       totalResParcels += 1               #use summarystatistics against tableSelRecs to get totals per USE_CATEGORY and write results to tableSumRecs               arcpy.Statistics_analysis(r'in_memory\tableSelRecs', r'in_memory\tableSumRecs', [["SHAPE_Area", "SUM"]], "USE_CATEGORY")           with arcpy.da.SearchCursor(r'in_memory\tableSumRecs',['USE_CATEGORY','SUM_SHAPE_Area']) as selCur:               #list to store calculation for each USE_CATEGORY value used in DI calculation               interimValue = [0]               #list to store calculation for each USE_CATEGORY value used in RDI calculation               interimResValue = [0]               for sRow in selCur:                   #divide the USE_CATEGORY area by the total community area then square that number & append it to the list                   interimValue.append(np.square(sRow[1]/totalArea))                   if sRow[0] in('Residential - Low Density','Residential - Medium Density','Residential - High Density'):                       interimResValue.append(np.square(sRow[1]/totalResArea))               #sum the values in the list then subtract that value from 1 to get the DI for the selected blocks               DI = 1 - sum(interimValue)               #sum the values in the list then subtract that value from 1 to get the RDI for the selected blocks               RDI = 1 - sum(interimResValue)               parameters[1].value = DI               parameters[2].value = RDI               parameters[3].value = totalParcels               parameters[4].value = totalArea               parameters[5].value = totalResParcels               parameters[6].value = totalResArea           #Delete in memory tables           arcpy.Delete_management(r'in_memory\tableSelRecs')           arcpy.Delete_management(r'in_memory\tableSumRecs')         except arcpy.ExecuteError:           print(arcpy.GetMessages(2)) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ ... View more

When you say it takes 1.5 seconds each time, is that the time when freshly initialized and ran, or does it take that long even if run successively in the same process/session? It does run a little faster on subsequent runs. Depending on which ArcPy components are being used, initializing/loading all of them into a clean Python process/session make take a second.  You might be seeing the same times between 20 and 5000 parcels because the tool runs so quick that the time you are seeing is the overhead of initializing objects and libraries. I think you are correct. If you are going to time your functions, it might be worth timing how long it takes for your import statements as well. Good point, would be interesting to see. Thanks Joshua ... View more

Why is there a need to write data to the in_memory tables? Seems like the stats you are calculating only rely on read data. Perhaps a dictionary could serve the purpose of you intermediate in_memory tables? I believe when I first started working on this script (and created a different geonet thread) the suggestions were to use intermediate tables with summary statistics or use a dictionary. I got the script working with summary statistics (with a plan later to try to get it working with a dictionary but as yet haven't got back to it). Significant bottlenecks I see are: 1. Creating the intermediate tables, adding fields to them, and inserting values Agreed. 2. Writing data to numpy arrays from the cursors. Seems like the np.square()  calculations are entirely doable outside of numpy. Certainly more convenient in np though. Agree on both points. 3. Using arcpy.mapping to get a hook into the layer. Maybe you could just have a toolbox w/ an input feature layer variable that had a default value set to your desired layer name? This is a change that I have made as I had the python toolbox published as a gp service to use as a geoprocessing widget in web appbuilder and without inputs and outputs it's basically useless. So making this change of using inputs/outputs and removing the need to hook into the layer has brought the script run time down to 0.3 - 0.4 seconds. Thanks for your input Chris ... View more

And a little further reading and I'm starting to think that I need to rework my script so that it has inputs and outputs. ... View more

I think the reason that I don't see "Set as Input of Geoprocessing" as an available option for me is because my script has no inputs. In the script I am just using a searchcursor on the landuse layer to work with selected records. But I'm starting to think that in the web appbuilder world it's not that simple. ... View more

Yes, I just a few minutes figured out the bit about the ‘CURRENT’. I did see recently when looking at the documentation for the Select widget that you could specify the results of the selection to be used as input to a gp widget (I think). But I do not see that available in the web appbuilder I see when I log into our agol. Thanks Jake ... View more

Here is the error message provided when message level = info: Running script DiversityIndex... Traceback (most recent call last): File "\\coc\apps\gis\gistools\gp\2.7\dev\int_planning\diversityindex\modules\DiversityIndex_Mod.py", line 82, in execute mxd = arcpy.mapping.MapDocument('CURRENT') File "c:\program files\arcgis\server\arcpy\arcpy\arcobjects\mixins.py", line 612, in __init__ super(MapDocumentMethods, self).__init__(mxd) File "c:\program files\arcgis\server\arcpy\arcpy\arcobjects\_base.py", line 47, in __init__ for arg in args)) RuntimeError: Object: CreateObject cannot open map document Failed to execute (DiversityIndex). ... View more

Hi Jake, thanks for the feedback. I will check the message level and set it to Info. And with the service, I was 'hoping' that since the script in arcmap is told which layer to look at for selected blocks, that that would carry over to the execution of the gp service. ... View more

So I talked to our server guys as the plan is to have this script published as a gp service which I can then use in web appbuilder. They told me that 1.5 seconds is very good. So maybe I should move forward with having the script published. I'm thankful for the comments here as I have taken away some good learnings and some pointers to help me improve the script. thanks ... View more

Ok, thanks Dan. I see how that could help me identify any bottleneck. ... View more

I have no idea what a decorator is. ... View more

A good point. It just seems odd to me that it's no faster when run with a small number of parcels. ... View more

Thanks Dan for the comments and suggestions. ... View more