Latest Contributions by JoshuaBixby

This is the third in a multi-part series on ArcPy cursors; particularly, working with ArcPy cursors as iterable objects in Python.  The first part in the series looks at some of the important components of iteration in Python.  The second part in the series looks at iterating and looping over ArcPy Data Access cursors.  The third part in the series looks at using several Python built-in and itertool functions with ArcPy Data Access cursors.  The fourth part in the series will look at using generators or generator expressions to separate selection or filtering logic for code re-use.  Fifth or following parts are unknown at this point. The first two parts in this series cover iteration components of Python and iterating/looping in Python, both of which are crucial to understanding and working with iterables.  Beyond manually iterating or looping over an iterable, there are numerous Python built-in functions that work with iterables and sequences.  Starting back in version 2.3, an itertools module was introduced that contains "functions creating iterators for efficient looping." The focus of this series is treating ArcPy Data Access cursors as Python iterables.  Like most things in life, there is more than one way to answer a question using ArcGIS, and I want to provide some contrasting examples along with Pythonic examples.  As much as I like writing idiomatic Python, there will be plenty of times when using native geoprocessing tools will outperform straight Python.  Writing fast code is great, but it's a separate discussion for a different day. There are so many built-in and itertool functions that work with iterables, I can't possibly demonstrate them all, but I will demonstrate a handful to illustrate how such functions work nicely with ArcPy Data Access cursors.  For this and future parts in the series, I will leave the previous examples behind in favor of a real-world dataset that readers can download and experiment with themselves.  Specifically, I will use the USA States layer package included with Esri Data & Maps and available on ArcGIS.com. The same ArcPy Data Access SearchCursor will be used for most of the examples below: >>> layer = r'USA States\USA States (below 1:3m)' >>> fields = ["STATE_ABBR", "SHAPE@AREA", "SUB_REGION", "POP2010"] >>> cursor = arcpy.da.SearchCursor(layer, fields) >>> ‍‍‍‍ One question that comes up from time to time in the forums/GeoNet is how to count the number of records in a data set or selection set using cursors: >>> # Example 1: Get record count using ArcGIS geoprocessing tool >>> arcpy.GetCount_management(layer) <Result '52'> >>> >>> # Example 2: Get record count using variable as counter >>> with cursor: ...     i = 0 ...     for row in cursor: ...         i = i + 1   # or i += 1 ...     print i ... 52 >>> >>> # Example 3: Get record count using built-in list and len functions >>> with cursor: ...     print len(list(cursor)) ... 52 >>> >>> # Example 4: Get record count using built-in sum function >>> with cursor: ...     print sum(1 for row in cursor) ... 52 >>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Looking over the record counting examples: Example 1 uses ArcGIS's Get Count geoprocessing tool. The Get Count tool retrieves the number of records in a feature class, table, layer, or raster; it does not operate against cursors.  It can also be used in the GUI. Example 2 loops over the cursor using a variable as a counter. A counting loop structure is common across a wide range of programming languages. Example 3 uses built-in list and len functions. The list function converts the entire cursor into a Python list, and the len function returns the length of the list, which corresponds to the number of records in the cursor. Creating a Python list does require copying the entire contents of the cursor into memory, which could become an issue for extremely large data sets. Example 4 uses the built-in sum function along with a generator expression. Using a Python generator expression instead of a list does not copy the entire contents of the cursor into memory. I included Example 1 because it is the highest performing approach to getting record counts of data sets and selection sets within the ArcGIS framework, but it doesn't deal with a cursor as a Python iterable, which is the focus of this series.  Example 2 is functionally and syntactically correct, although I would argue it isn't the most Pythonic.  Examples 3 and 4 both use Python built-in functions that treat a cursor as an iterable, but it is arguable whether Example 3 or 4 is more Pythonic because each approach has strengths and weaknesses. Another question that comes up occasionally is how to retrieve a record from a data set based on the minimum or maximum values of one of the fields in the data set.  This next set of examples will retrieve the record/row for the state with the highest population in 2010 (POP2010): >>> # Retrieve table name for data source of layer >>> desc = arcpy.Describe(layer) >>> fc_name = desc.featureClass.baseName >>> >>> # Example 5: Get maximum population record using ArcGIS geoprocessing tools >>> summary_table = arcpy.Statistics_analysis(layer, ...                                           "in_memory/summary_max", ...                                           "POP2010 MAX") ...                                      >>> arcpy.AddJoin_management(layer, ...                          "POP2010", ...                          summary_table, ...                          "MAX_POP2010", ...                          "KEEP_COMMON") ...                     >>> joined_fields = [(field if "@" in field else ".".join([fc_name, field])) ...                  for field ...                  in fields] >>> cursor_sql_join = arcpy.da.SearchCursor(layer, joined_fields) >>> print next(cursor_sql_join) (u'CA', 41.639274447708424, u'Pacific', 37253956) >>> del cursor_sql_join >>> arcpy.RemoveJoin_management(layer, "summary_max") <Result 'USA States\\USA States (below 1:3m)'> >>> >>> # Example 6: Get maximum population record using SQL subquery >>> sql = "POP2010 IN ((SELECT MAX(POP2010) FROM {}))".format(fc_name) >>> cursor_sql_subqry = arcpy.da.SearchCursor(layer, fields, sql)                                >>> print next(cursor_sql_subqry) (u'CA', 41.639274447708424, u'Pacific', 37253956) >>> del cursor_sql_subqry >>> >>> # Example 7: Get maximum population record by looping and comparing >>> with cursor: ...     max_row = next(cursor) ...     for row in cursor: ...         if row[3] > max_row[3]: ...             max_row = row ...        >>> print max_row (u'CA', 41.639274447708424, u'Pacific', 37253956) >>> >>> # Example 8: Get maximum population record using built-in max function >>> from operator import itemgetter >>> with cursor: ...     print max(cursor, key=itemgetter(3)) ... (u'CA', 41.639274447708424, u'Pacific', 37253956) >>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Looking over the maximum record examples: Examples 5 and 6 require the table name for the data source of the layer. The table name was found using the ArcPy Describe function and properties of two different Describe objects (the property lookups were chained together on line 03) Examples 5 and 6 both need the underlying table name but for different reasons. With Example 5, the base table name is needed to re-create the cursor after the summary statistics table is joined to the layer's source table. With Example 6, the base table name is needed to create the subquery used in the cursor's SQL WHERE clause. Example 5 uses ArcGIS's Summary Statistics and Add Join geoprocessing tools. The Summary Statistics tool finds the maximum population value in the dataset.  The Add Join tool allows the result of the Summary Statistics tool to be linked back to the original layer to find the corresponding record with the maximum population. Before creating a cursor against the newly joined layer, the original fields need to be updated to prepend the table name to the field name (line 16). Example 6 uses an SQL subquery in the WHERE clause of the SQL query. The SQL subquery identifies the maximum population value.  The value(s) returned from the SQL subquery are used to select record(s) in the original data set. Example 7 loops over the cursor using a variable to hold the record with the maximum value. A loop structure is common across a wide range of programming languages. Example 8 uses the built-in max function along with the operator.itemgetter function. The max function operates on an iterable, but it is necessary to use operator.itemgetter since the cursor represents an iterable of tuples and not an iterable of numeric data types. I included Example 5 because it only uses ArcGIS geoprocessing tools and can be implemented in the GUI.  Although it can be implemented in the GUI with no scripting skills, Example 5 is also the most cumbersome, i.e., it has the most steps, is the slowest, and creates intermediate products.  With just a little bit of SQL or Python knowledge, the doors open to more eloquent and higher performing approaches.  Similar to Example 2 above, Example 7 is functionally and syntactically correct, although I would argue Example 8 is more Pythonic. Instead of getting just the State with the largest population in 2010, let's print States and their populations by descending population: >>> # Example 9: Print State and population by descending population >>> #            using Sort geoprocessing tool >>> sorted_table = arcpy.Sort_management(layer, ...                                      "in_memory/sorted_pop", ...                                      "POP2010 DESCENDING") ...                                      >>> cursor_sorted_table = arcpy.da.SearchCursor(sorted_table, fields) >>> with cursor_sorted_table: ...     for state, area, sub_region, pop2010 in cursor_sorted_table: ...         print "{}, {}".format(state, pop2010) ...         CA, 37253956 TX, 25145561 NY, 19378102 ... DC, 601723 WY, 563626 PR, -99 >>> del cursor_sorted_table >>> >>> # Example 10: Print State and population by descending population >>> #             appending SQL ORDER BY clause >>> sql = "ORDER BY POP2010 DESC" >>> cursor_orderby_sql = arcpy.da.SearchCursor(layer, fields, sql_clause=(None, sql)) >>> with cursor_orderby_sql: ...     for state, area, sub_region, pop2010 in cursor_orderby_sql: ...         print "{}, {}".format(state, pop2010) ...         CA, 37253956 TX, 25145561 NY, 19378102 ... DC, 601723 WY, 563626 PR, -99 >>> del cursor_orderby_sql >>> >>> # Example 11: Print State and population by descending population >>> #             using built-in sorted function >>> with cursor: ...     for state, area, sub_region, pop2010 in sorted(cursor, ...                                                    key=itemgetter(3), ...                                                    reverse=True): ...         print "{}, {}".format(state, pop2010) ...         CA, 37253956 TX, 25145561 NY, 19378102 .... DC, 601723 WY, 563626 PR, -99 >>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Looking over the descending population examples: Example 9 uses ArcGIS's Sort goeprocessing tool. The Sort tool sorts the original table into a new table.  A new cursor needs to be defined using the newly sorted feature class. Example 10 uses an SQL ORDER BY clause. The SQL ORDER BY clause is passed as part of the sql_clause while creating a new cursor. Example 11 uses the built-in sorted function along with the operator.itemgetter function. The sorted function operates on an iterable, but it is necessary to use operator.itemgetter since the cursor represents an iterable of tuples and not an iterable of numeric data types. The sorted function converts the entire cursor into a Python list, and creating a Python list does require copying the entire contents of the cursor into memory, which could become an issue for extremely large data sets. I included Example 9 because it uses ArcGIS geoprocessing tools and can be implemented easily in the GUI.  Similar to Example 5 above, Example 9 is cumbersome if one is scripting instead of using the GUI.  Example 10 uses some basic SQL for a straightforward solution, although it does involve having to create another cursor instead of recycling the existing cursor.  Example 11 is idiomatic in that it uses the built-in sorted function and treats the cursor as an iterable.  Since sorted does return a newly sorted Python list from an iterable, using the function could become an issue with extremely large data sets. There are numerous other examples I thought up, but this post is already longer than I expected.  I believe the 3 sets of examples above demonstrate how Python built-in functions that operate on iterables can be used with ArcPy Data Access cursors to write idiomatic Python for ArcGIS.  The next part of the series looks at using generators and generator expressions with ArcPy Data Access cursors. ... View more

This is the second in a multi-part series on ArcPy cursors; particularly, working with ArcPy cursors as iterable objects in Python.  The first part in the series looks at some of the important components of iteration in Python.  The second part in the series looks at iterating and looping over ArcPy Data Access cursors.  The third part in the series looks at using several Python built-in and itertool functions with ArcPy Data Access cursors.  The fourth part in the series will look at using generators or generator expressions to separate selection or filtering logic for code re-use.  Fifth or following parts are unknown at this point. The first part in this series is likely a bit academic for some ArcPy scripters, but I believe a little theory goes a long ways to understanding the practice of something, in this case working with ArcPy Data Access cursors in an idiomatic way.  Also, the terms and concepts laid out in the first post will come up time and again throughout the series. Before moving on, I will put a plug in for a presentation from a few years back: Loop Like A Native.  Ned Batchelder gives a nice overview of looping in Python, especially for those with looping experience in other programming languages.  It took me several times of watching it for the whole presentation to sink in, but it really did change the way I view iterating and looping in Python. Recycling the Python list and ArcPy cursor examples from the previous post, let's call iter() to return an iterator for manually stepping through each iterable. >>> #create list, attach 2 iterators, and retrieve values by >>> #    calling next() and object.next() >>> l = [10, 20, 30, 40, 50] >>> it_l = iter(l) >>> it2_l = iter(l) >>> print next(it_l), next(it2_l) 10 10 >>> print it_l.next(), it2_l.next() 20 20 >>> >>> #create search cursor, attach 2 iterators, and retrieve values by >>> #    calling next and object.next() >>> cur = arcpy.da.SearchCursor(fc, ["OID@", "SHAPE@"]) >>> it_cur = iter(cur) >>> it2_cur = iter(cur) >>> print next(it_cur), next(it2_cur) (1, <Polyline object at 0x1100fcf0[0x1100ff20]>) (2, <Polyline object at 0x1100fcf0[0x1100ff20]>) >>> print it_cur.next(), it2_cur.next() (3, <Polyline object at 0x1100fcf0[0x1100ff20]>) (4, <Polyline object at 0x1100fcf0[0x1100ff20]>) >>> print next(cur) (5, <Polyline object at 0x1100fcf0[0x1100ff20]>) >>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ There is a fair amount to comment on with the code above: A single iterable can have multiple iterators simultaneously accessing it.  How the iterable behaves with multiple iterators is implementation specific. As the iterator definition states in the Python Glossary, a list "produces a fresh new iterator each time you pass it to a iter() function or use it in a for loop."  This explains why lines 06-07 and 08-09 are printing out the same values for both iterators. In contrast to the Python list, the ArcPy Data Access search cursor does not produce a new iterator, i.e., each subsequent call to an iter() function returns the same iterator object that is already in use.  In these types of situations, each call to next() moves the iterator ahead one element regardless of which iterator makes the call.  This explains why lines 16-17 show the first and second OID instead of showing the first OID twice. An iterator can be moved ahead by using either the built-in next() function or the object's next method.  Starting at Python 3.0, with the adoption of PEP 3114, the preferred method to manually iterate is the built-in next() function. Since ArcPy Data Access cursors are their own iterator, one doesn't need to call iter() to get an iterator object before calling next().  Line 20 shows the cursor object itself can be passed to next() to retrieve the next item and move the cursor ahead. Fortunately for us, the Python for statement does a lot of lifting to streamline the steps so we don't have to manually retrieve an iterator and call next() until the end of the iterable is reached. Revisiting the SearchCursor documentation: Summary SearchCursor establishes read-only access to the records returned from a feature class or table. Returns an iterator of tuples. The order of values in the tuple matches the order of fields specified by the field_names argument. Discussion Geometry properties can be accessed by specifying the token SHAPE@ in the list of fields. Search cursors can be iterated using a For loop.  [Removed at 10.3.1:  Search cursors also support With statements; using a With statement will guarantee close and release of database locks and reset iteration]. As one can see, the documentation clearly states the arcpy.da.SearchCursor returns an iterator, an iterator of tuples to be specific.  It makes sense to have it return tuples versus lists since we are using a search cursor that can't update data and tuples are immutable by design.  The second statement in the Discussion section is redundant since for in Python iterates over any iterable, SearchCursor or otherwise, but that statement wouldn't stand out as being redundant if Esri didn't remove the rest of the paragraph that used to follow. This is a not-so quick aside on an Esri #fail, an example of how not to handle customer feedback. As shown above, prior to ArcGIS 10.3.1, Esri included a couple statements regarding Python with statements.  The fact that ArcPy Data Access cursors support with statements is worth pointing out, even documenting one might say.  The issue with the two statements was really just an issue with the latter statement, I guarantee it!  Guarantee is a strong word, a definitive word, and the problem is not all database locks are closed and released. A bug was submitted for the documentation to be updated, BUG-000083762: In each cursor documentation, specify the type of lock being closed and released, as a shared lock is still present in the geodatabase after the 'with' statement executes.  The issue was identified as "fixed" in ArcGIS 10.3.1.  If you want to go find that clarification on locks, I already showed it to you.  Yep, there isn't any, they simply removed the statement about locks.  The insult to injury, they also removed a very important statement about Data Access cursors supporting the Python with statement. Although the documentation speaks to iterators and iterating, I feel a real opportunity was lost with the code samples to demonstrate a handy Python feature.  A lot of ArcGIS users that are new to Python learn the language by emulating code examples.  In terms of showing Pythonic examples, the ArcPy Data Access cursors are a mixed bag. Code Sample SearchCursor example 1 Use SearchCursor to step through a feature class and print specific field values and the x,y coordinates of the point. import arcpy fc = 'c:/data/base.gdb/well' fields = ['WELL_ID', 'WELL_TYPE', 'SHAPE@XY'] # For each row print the WELL_ID and WELL_TYPE fields, and the # the feature's x,y coordinates with arcpy.da.SearchCursor(fc, fields) as cursor:     for row in cursor:         print('{0}, {1}, {2}'.format(row[0], row[1], row[2])) As pointed out in my earlier soapbox/aside, the ArcPy Data Access documentation fails to mention that cursors support the Python with statement.  That said, support is implied by the use of Python with statements in the examples.  It is worth one's time to read up on Python with statements, and I encourage their use with ArcPy Data Access cursors whenever possible. Whereas the documentation examples demonstrate using Python with statements, even though the documentation itself doesn't state they are supported, the examples do fail to demonstrate the use of iterable or sequence unpacking.  Iterable or sequence unpacking is a great feature of Python, and it can be used to make code much more compact and readable at the same time.  Sequence unpacking is briefly mentioned in the Python documentation for Tuples and Sequences, and PEP 3132 -- Extended Iterable Unpacking discusses changes introduced in Python 3.0. Let's take a look at how iterable unpacking can be used with SearchCursor example 1 from above. import arcpy fc = 'c:/data/base.gdb/well' fields = ['WELL_ID', 'WELL_TYPE', 'SHAPE@XY' ] # For each row print the WELL_ID and WELL_TYPE fields, and the # the feature's x,y coordinates # Original example using sequence indexing with arcpy.da.SearchCursor(fc, fields) as cursor:     for row in cursor:         print('{0}, {1}, {2}'.format(row[0], row[1], row[2])) # Example using manual sequence unpacking with arcpy.da.SearchCursor(fc, fields) as cursor:     for row in cursor:         well_id = row[0]         well_type = row[1]         well_xy = row[2]         print('{0}, {1}, {2}'.format(well_id, well_type, well_xy)) # Example using built-in sequence unpacking with arcpy.da.SearchCursor(fc, fields) as cursor:     for well_id, well_type, well_xy in cursor:         print('{0}, {1}, {2}'.format(well_id, well_type, well_xy)) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ I provided an example of manual sequence unpacking because it is fairly common to see that pattern with people coming over to Python from other languages.  Although manual unpacking is syntactically and functionally correct, it can usually be replaced by using built-in unpacking, thus saving some lines of code and being more idiomatic.  With this specific series of examples, it turns out that using built-in sequence unpacking isn't any more compact than using sequence indexing; however, I find reading and maintaining code that uses sequence unpacking is much more straightforward than having to remember which index means what in a sequence. The Python for statement, with statement, and iterable/sequence unpacking; all essentials when working with the iterable cursor. ... View more