I have a parameter in a python toolbox which allows a user to select a field from a dataset given in a previous parameter. # Feature Class to absorb geometry from param2 = arcpy . Parameter ( displayName = "Geometry Feature Class" , name = "in_geoFC" , datatype = "GPFeatureLayer" , parameterType = "Required" , direction = "Input" ) # Table ID field param3 = arcpy . Parameter ( displayName = "Table Geometry ID Field" , name = "table_geoIDField" , datatype = "GPString" , parameterType = "Required" , direction = "Input" , enabled = False ) param3 . filter . type = "ValueList" param3 . filter . list = [ ] The updateParameters section contains logic to update Parameter 3 with the field names for all of the fields in the dataset provided in parameter 2 which have a datatype of string: def updateParameters ( self , parameters ) : """Modify the values and properties of parameters before internal validation is performed. This method is called whenever a parameter has been changed.""" # if 'in_geoFC' is populated with a value if parameters [ 2 ] . value : # if 'in_geoFC' does not have an error set if not parameters [ 2 ] . hasError ( ) : # Create a list of all of the fields in the 'in_geoFC' # which have a datatype of 'String' fc_geoIDFields = [ field . name for field in arcpy . Describe ( parameters [ 2 ] . valueAsText ) . fields if field . type == 'String' ] # Enable the parameter parameters [ 3 ] . enabled = True # Populate the parameter with the list of text fields in the # table parameters [ 3 ] . filter . list = fc_geoIDFields This all works just fine... There's one more thing I need to do though. Whichever field the user selects for parameter 3, I need to ensure that the values contained in this field are unique - every record must have a unique value. I know of a pretty easy and elegant way to get the number of unique values in that field: len ( set ( r [ 0 ] for r in arpy . da . SearchCursor ( parameters [ 1 ] . valueAsText , parameters [ 2 ] . valueAsText ) ) ) What I don't know is the quickest and most elegant way to get the total number of features in that dataset so that I can compare it to the number of unique values in that field and thus, determine if all of the values in that field are unique. Keep in mind that this would be occurring in the updateMessages function, so it needs to be a fairly quick process. Any thoughts on how to get a record count super fast?

Have you done any benchmarking? Get Count easily beats other methods for getting the total records in a data set, especially for larger data set. If you are already running a cursor against the data set, then using Get Count might be unnecessary, but any extra time added to the script would be from an necessary call and not a slow function. In terms of balancing simplicity and performance, I am a big fan of Counter, as I stated above. It is trivially slower than proposed set operations, and it gives you much richer information in case that might have value at some point. import collections def unique_check_cnt ( iterable ) : counter = collections . Counter ( iterable ) return True if counter . most_common ( 1 ) [ 0 ] [ 1 ] == 1 else False fc = # path to feature class fld = # field name to check for uniqueness with arcpy . da . SearchCursor ( fc , fld ) as cur : print unique_check_cnt ( i for i , in cur ) If the data sets are large and there is a reasonable chance there will be duplicates, then it might pay off to implement a method that can stop as soon as a duplicate is found. import collections def unique_check_defdict ( iterable ) : d = collections . defaultdict ( int ) for i in iterable : if d [ i ] > 0 : return False d [ i ] += 1 else : return True fc = # path to feature class fld = # field name to check for uniqueness with arcpy . da . SearchCursor ( fc , fld ) as cur : print unique_check_defdict ( i for i , in cur )

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Checking if field is unique value field

JohnDye

I have a parameter in a python toolbox which allows a user to select a field from a dataset given in a previous parameter.

<SPAN class="comment token"># Feature Class to absorb geometry from</SPAN>
param2 <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>Parameter<SPAN class="punctuation token">(</SPAN>
&nbsp;&nbsp;&nbsp; displayName<SPAN class="operator token">=</SPAN><SPAN class="string token">"Geometry Feature Class"</SPAN><SPAN class="punctuation token">,</SPAN>
&nbsp;&nbsp;&nbsp; name<SPAN class="operator token">=</SPAN><SPAN class="string token">"in_geoFC"</SPAN><SPAN class="punctuation token">,</SPAN>
&nbsp;&nbsp;&nbsp; datatype<SPAN class="operator token">=</SPAN><SPAN class="string token">"GPFeatureLayer"</SPAN><SPAN class="punctuation token">,</SPAN>
&nbsp;&nbsp;&nbsp; parameterType<SPAN class="operator token">=</SPAN><SPAN class="string token">"Required"</SPAN><SPAN class="punctuation token">,</SPAN>
&nbsp;&nbsp;&nbsp; direction<SPAN class="operator token">=</SPAN><SPAN class="string token">"Input"</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍

<SPAN class="comment token"># Table ID field</SPAN>
param3 <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>Parameter<SPAN class="punctuation token">(</SPAN>
&nbsp;&nbsp;&nbsp; displayName<SPAN class="operator token">=</SPAN><SPAN class="string token">"Table Geometry ID Field"</SPAN><SPAN class="punctuation token">,</SPAN>
&nbsp;&nbsp;&nbsp; name<SPAN class="operator token">=</SPAN><SPAN class="string token">"table_geoIDField"</SPAN><SPAN class="punctuation token">,</SPAN>
&nbsp;&nbsp;&nbsp; datatype<SPAN class="operator token">=</SPAN><SPAN class="string token">"GPString"</SPAN><SPAN class="punctuation token">,</SPAN>
&nbsp;&nbsp;&nbsp; parameterType<SPAN class="operator token">=</SPAN><SPAN class="string token">"Required"</SPAN><SPAN class="punctuation token">,</SPAN>
&nbsp;&nbsp;&nbsp; direction<SPAN class="operator token">=</SPAN><SPAN class="string token">"Input"</SPAN><SPAN class="punctuation token">,</SPAN>
&nbsp;&nbsp;&nbsp; enabled<SPAN class="operator token">=</SPAN><SPAN class="token boolean">False</SPAN><SPAN class="punctuation token">)</SPAN>
param3<SPAN class="punctuation token">.</SPAN>filter<SPAN class="punctuation token">.</SPAN>type <SPAN class="operator token">=</SPAN> <SPAN class="string token">"ValueList"</SPAN>
param3<SPAN class="punctuation token">.</SPAN>filter<SPAN class="punctuation token">.</SPAN>list <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

The updateParameters section contains logic to update Parameter 3 with the field names for all of the fields in the dataset provided in parameter 2 which have a datatype of string:

<SPAN class="keyword token">def</SPAN> <SPAN class="token function">updateParameters</SPAN><SPAN class="punctuation token">(</SPAN>self<SPAN class="punctuation token">,</SPAN> parameters<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="string token">"""Modify the values and properties of parameters before internal
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; validation is performed.&nbsp; This method is called whenever a parameter
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; has been changed."""</SPAN>
&nbsp;&nbsp;&nbsp; 
&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># if 'in_geoFC' is populated with a value</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> parameters<SPAN class="punctuation token">[</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>value<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># if 'in_geoFC' does not have an error set</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> <SPAN class="operator token">not</SPAN> parameters<SPAN class="punctuation token">[</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>hasError<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token">#&nbsp; Create a list of all of the fields in the 'in_geoFC'</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># which have a datatype of 'String'</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fc_geoIDFields <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>field<SPAN class="punctuation token">.</SPAN>name <SPAN class="keyword token">for</SPAN> field <SPAN class="keyword token">in</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>Describe<SPAN class="punctuation token">(</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parameters<SPAN class="punctuation token">[</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>valueAsText<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>fields
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> field<SPAN class="punctuation token">.</SPAN>type <SPAN class="operator token">==</SPAN> <SPAN class="string token">'String'</SPAN><SPAN class="punctuation token">]</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># Enable the parameter</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parameters<SPAN class="punctuation token">[</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>enabled <SPAN class="operator token">=</SPAN> <SPAN class="token boolean">True</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># Populate the parameter with the list of text fields in the</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="comment token"># table</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parameters<SPAN class="punctuation token">[</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>filter<SPAN class="punctuation token">.</SPAN>list <SPAN class="operator token">=</SPAN> fc_geoIDFields‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

This all works just fine...

There's one more thing I need to do though. Whichever field the user selects for parameter 3, I need to ensure that the values contained in this field are unique - every record must have a unique value.

I know of a pretty easy and elegant way to get the number of unique values in that field:

len<SPAN class="punctuation token">(</SPAN>set<SPAN class="punctuation token">(</SPAN>r<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="keyword token">for</SPAN> r <SPAN class="keyword token">in</SPAN> arpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>parameters<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>valueAsText
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="punctuation token">,</SPAN> parameters<SPAN class="punctuation token">[</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>valueAsText<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

What I don't know is the quickest and most elegant way to get the total number of features in that dataset so that I can compare it to the number of unique values in that field and thus, determine if all of the values in that field are unique.

Keep in mind that this would be occurring in the updateMessages function, so it needs to be a fairly quick process.

Any thoughts on how to get a record count super fast?

Najděte více příspěvků označených jako

Accepted answers

JoshuaBixby

Have you done any benchmarking? Get Count easily beats other methods for getting the total records in a data set, especially for larger data set. If you are already running a cursor against the data set, then using Get Count might be unnecessary, but any extra time added to the script would be from an necessary call and not a slow function.

In terms of balancing simplicity and performance, I am a big fan of Counter, as I stated above. It is trivially slower than proposed set operations, and it gives you much richer information in case that might have value at some point.

<SPAN class="keyword token">import</SPAN> collections

<SPAN class="keyword token">def</SPAN> <SPAN class="token function">unique_check_cnt</SPAN><SPAN class="punctuation token">(</SPAN>iterable<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; counter <SPAN class="operator token">=</SPAN> collections<SPAN class="punctuation token">.</SPAN>Counter<SPAN class="punctuation token">(</SPAN>iterable<SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> <SPAN class="token boolean">True</SPAN> <SPAN class="keyword token">if</SPAN> counter<SPAN class="punctuation token">.</SPAN>most_common<SPAN class="punctuation token">(</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">==</SPAN> <SPAN class="number token">1</SPAN> <SPAN class="keyword token">else</SPAN> <SPAN class="token boolean">False</SPAN>
&nbsp;&nbsp;&nbsp; 
fc <SPAN class="operator token">=</SPAN> <SPAN class="comment token"># path to feature class</SPAN>
fld <SPAN class="operator token">=</SPAN> <SPAN class="comment token"># field name to check for uniqueness</SPAN>

<SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>fc<SPAN class="punctuation token">,</SPAN> fld<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> cur<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">print</SPAN> unique_check_cnt<SPAN class="punctuation token">(</SPAN>i <SPAN class="keyword token">for</SPAN> i<SPAN class="punctuation token">,</SPAN> <SPAN class="keyword token">in</SPAN> cur<SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

If the data sets are large and there is a reasonable chance there will be duplicates, then it might pay off to implement a method that can stop as soon as a duplicate is found.

<SPAN class="keyword token">import</SPAN> collections

<SPAN class="keyword token">def</SPAN> <SPAN class="token function">unique_check_defdict</SPAN><SPAN class="punctuation token">(</SPAN>iterable<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; d <SPAN class="operator token">=</SPAN> collections<SPAN class="punctuation token">.</SPAN>defaultdict<SPAN class="punctuation token">(</SPAN>int<SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">for</SPAN> i <SPAN class="keyword token">in</SPAN> iterable<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> d<SPAN class="punctuation token">[</SPAN>i<SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">></SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> <SPAN class="token boolean">False</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; d<SPAN class="punctuation token">[</SPAN>i<SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">+=</SPAN> <SPAN class="number token">1</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">else</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> <SPAN class="token boolean">True</SPAN>

fc <SPAN class="operator token">=</SPAN> <SPAN class="comment token"># path to feature class</SPAN>
fld <SPAN class="operator token">=</SPAN> <SPAN class="comment token"># field name to check for uniqueness</SPAN>

<SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>fc<SPAN class="punctuation token">,</SPAN> fld<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> cur<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">print</SPAN> unique_check_defdict<SPAN class="punctuation token">(</SPAN>i <SPAN class="keyword token">for</SPAN> i<SPAN class="punctuation token">,</SPAN> <SPAN class="keyword token">in</SPAN> cur<SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

All comments

DanPatterson_Retired

Your call on answer or useful...

As for speed, if python toolbox functionality isn't needed, why use them? Conventional toolboxes are simple and clean and I like the fact that the 'human' interface actually has to think about what they are doing, instead of writing tools that account for all issues between the keyboard and the chair

JohnDye

That could be an option. I'm not sure how it would perform though given that our internal network is absolute garbage.

In any case, once I implemented the custom function I had defined in the updateMessages section of my toolbox - for some reason that is not clear to me, the result is almost instantaneous. Must faster than the 6 seconds I was getting back when running this in the Python Window.

Pretty much every answer on this post is correct though. Which one to mark correct? I suppose I'll defer to the Dan the Man ( Dan Patterson‌ ) to pick the best answer.

On a side note - why would a Python Toolbox be able to run that function so much faster than the Python Window? I get that Python Toolboxes can be background enabled (which this one is), allowing them to run in a separate, 64-bit thread - but the performance difference is so dramatic (I seriously mean that the error is raised within 1 second) that I feel like there has to be something else going on that I just don't know about.

JamesCrandall

Is this an SDE feature class? If so, have you tried a standard SQL count statement against the RBMS?

JohnDye

Would be nice if there were a Describe property for record count. That would be fast. Seems strange to me that there's not. FIDSet on a layer will return selected records, but you have to set up a cursor and iterate the dataset just to get the full count.

DanPatterson_Retired

_as_narray ... I have used it recently in some of my blogs.

<SPAN class="keyword token">import</SPAN> numpy <SPAN class="keyword token">as</SPAN> np

a <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>in_fc<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'OID@'</SPAN><SPAN class="punctuation token">,</SPAN> explode_to_points<SPAN class="operator token">=</SPAN><SPAN class="token boolean">True</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>_as_narray<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>

<SPAN class="comment token"># in_fc is a featureclass,&nbsp; explode to points not needed, but I wanted a bigger file</SPAN>

len<SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">)</SPAN> <SPAN class="operator token">==</SPAN> len<SPAN class="punctuation token">(</SPAN>np<SPAN class="punctuation token">.</SPAN>unique<SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="token boolean">True</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Now if you are playing Code Golf, you can put that all in two lines

JohnDye

GetCount_management is a no-go because it has to instantiate the geoprocessor. Big performance hit. Not a good thing to do in Python Toolbox parameter validation unless you like making your users watch the cursor spin.

JamesCrandall

<SPAN class="keyword token">if</SPAN> int<SPAN class="punctuation token">(</SPAN>arcpy<SPAN class="punctuation token">.</SPAN>GetCount_management<SPAN class="punctuation token">(</SPAN>TheInputFeatureClass<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>getOutput<SPAN class="punctuation token">(</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="operator token"><></SPAN> len<SPAN class="punctuation token">(</SPAN>set<SPAN class="punctuation token">(</SPAN>arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>TheInputFeatureClass<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'TheFieldToEvaluate'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">print</SPAN> <SPAN class="string token">'length did not match'</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

JohnDye

numpy or pandas is likely going to be faster. I always forget about Pandas being available now. I wouldn't hold the import times against them with a Python toolbox. You pay those costs upfront at toolbox initialization unless you're importing inside of your tool classes.

JohnDye

I am doing that to determine the unique values. The catch is determining if that number of unique values matches the length of the recordset - which tells me every value in the given field is unique

JamesCrandall

Yep. Great point considering the OP needs this to perform during an onLoad event.

JamesCrandall

Considering Dan's valid point about numpy setup time, what about simply setting a "set" to the SearchCursor?

<SPAN class="keyword token">print</SPAN> len<SPAN class="punctuation token">(</SPAN>set<SPAN class="punctuation token">(</SPAN>arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>TheInputFeatureClass<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'TheFieldToEvaluate'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN>

DanPatterson_Retired

the timing for both conversion to numpy array has a bit of overhead but insignificant, for about 25,000 unique points, but still within a coffee sip and an import

%timeit len(np.unique(arcpy.da.FeatureClassToNumPyArray(in_fc, 'OID@')))
107 ms ± 2.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

JohnDye

Look at you rockstars! Lightning quick to respond.

I actually came up with this a few minutes later. Benchmarked it against 465,051 records. I'd like to get it down to 3 seconds or less

<SPAN class="keyword token">def</SPAN> <SPAN class="token function">calculateRunTime</SPAN><SPAN class="punctuation token">(</SPAN>function<SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">*</SPAN>args<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; startTime <SPAN class="operator token">=</SPAN> time<SPAN class="punctuation token">.</SPAN>time<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; result <SPAN class="operator token">=</SPAN> function<SPAN class="punctuation token">(</SPAN><SPAN class="operator token">*</SPAN>args<SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> time<SPAN class="punctuation token">.</SPAN>time<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="operator token">-</SPAN> startTime<SPAN class="punctuation token">,</SPAN> result
&nbsp;&nbsp;&nbsp; 
<SPAN class="keyword token">def</SPAN> <SPAN class="token function">isUniqueValueField</SPAN><SPAN class="punctuation token">(</SPAN>dataset<SPAN class="punctuation token">,</SPAN> field<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; idList <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>dataset<SPAN class="punctuation token">,</SPAN> field<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; idList<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> len<SPAN class="punctuation token">(</SPAN>idList<SPAN class="punctuation token">)</SPAN> <SPAN class="operator token">!=</SPAN> len<SPAN class="punctuation token">(</SPAN>set<SPAN class="punctuation token">(</SPAN>idList<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> <SPAN class="token boolean">False</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">else</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> <SPAN class="token boolean">True</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> calculateRunTime<SPAN class="punctuation token">(</SPAN>isUniqueValueField
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="punctuation token">,</SPAN> parameters<SPAN class="punctuation token">[</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>valueAsText
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="punctuation token">,</SPAN> parameters<SPAN class="punctuation token">[</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>valueAsText<SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">8.847000122070312</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="token boolean">True</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

JamesCrandall

<SPAN class="keyword token">import</SPAN> numpy <SPAN class="keyword token">as</SPAN> np

nparr <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>FeatureClassToNumPyArray<SPAN class="punctuation token">(</SPAN>TheInputFeatureClass<SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'TheFieldToEvaluate'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN>
uniqueValueCount <SPAN class="operator token">=</SPAN> len<SPAN class="punctuation token">(</SPAN>np<SPAN class="punctuation token">.</SPAN>unique<SPAN class="punctuation token">(</SPAN>nparr<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN> uniqueValueCount<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

DanPatterson_Retired

some may offer numpy and pandas solutions, but the initial load time

<SPAN class="operator token">%</SPAN>timeit len<SPAN class="punctuation token">(</SPAN>set<SPAN class="punctuation token">(</SPAN>random<SPAN class="punctuation token">.</SPAN>randint<SPAN class="punctuation token">(</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="number token">10</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">for</SPAN> i <SPAN class="keyword token">in</SPAN> range<SPAN class="punctuation token">(</SPAN><SPAN class="number token">1000000</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="number token">1.5</SPAN> s ± <SPAN class="number token">11.5</SPAN> ms per loop <SPAN class="punctuation token">(</SPAN>mean ± std<SPAN class="punctuation token">.</SPAN> dev<SPAN class="punctuation token">.</SPAN> of <SPAN class="number token">7</SPAN> runs<SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN> loop each<SPAN class="punctuation token">)</SPAN>

<SPAN class="keyword token">import</SPAN> numpy <SPAN class="keyword token">as</SPAN> np
<SPAN class="operator token">%</SPAN>timeit len<SPAN class="punctuation token">(</SPAN>np<SPAN class="punctuation token">.</SPAN>unique<SPAN class="punctuation token">(</SPAN>np<SPAN class="punctuation token">.</SPAN>random<SPAN class="punctuation token">.</SPAN>randint<SPAN class="punctuation token">(</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">10</SPAN><SPAN class="punctuation token">,</SPAN> size<SPAN class="operator token">=</SPAN><SPAN class="number token">1000000</SPAN><SPAN class="punctuation token">,</SPAN> dtype<SPAN class="operator token">=</SPAN><SPAN class="string token">'l'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="number token">43.4</SPAN> ms ± <SPAN class="number token">281</SPAN> µs per loop <SPAN class="punctuation token">(</SPAN>mean ± std<SPAN class="punctuation token">.</SPAN> dev<SPAN class="punctuation token">.</SPAN> of <SPAN class="number token">7</SPAN> runs<SPAN class="punctuation token">,</SPAN> <SPAN class="number token">10</SPAN> loops each<SPAN class="punctuation token">)</SPAN>

<SPAN class="comment token"># ---- is a coffee sip worth the difference? ----</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

JoshuaBixby

Personally, I like Counter.

Připojte se k diskusi

Přihlášení členové mohou přispívat, sledovat aktualizace a další. Jste tu noví? Zaregistrujte si bezplatný účet.

Rychlé odkazy

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.