Some Python Snippets

19384
46
07-23-2014 08:03 PM

General snippets

Split path and filename
filePathName = 'C:/Folder/SubFolder/afile.ext'
filePath,fileName=os.path.split(filePathName)

# Results in:
filePath = 'C:/Folder/SubFolder'
fileName = 'afile.ext'


Check if field exists
def FieldExist(tbl, fieldname):
"""Check if a field exists, return boolean"""
return bool(arcpy.ListFields(tbl, fieldname))

# format a number to 3 digits (with leading zero's)
text = "%03d" % (value,)


or following the suggestion of Neil Ayres‌:

length = 7
value = 123
text ="{0}".format(value).zfill(length)

# or indicating the number of decimals:
length = 10
value = 123.4567890
decimals = 2
text = "{0}".format(round(value, decimals)).zfill(length)

Also have a look at this website: Python String Format Cookbook | mkaz.com  as was suggested by Anthony Giles in this thread: What is label expression for formatting a number to have thousands separator and decimals?

Working with dictionaries

# Create a dictionary:
myDict = dict()
myDict = {}


# Check if key exists
if myKey in myDict:
# do something


# Read the value corresponding to a key
if myKey in myDict:
myVal = myDict[myKey]


# add a new key, value pair:
myDict.update({aKey: aValue})


# update an existing key, value pair
if myKey in myDict:
myVal = statsDict[myKey]
# do something with value
myVal += 1
# update key, value pair in dictionary
myDict.update({myKey: myVal})

# better and shorter is:
if myKey in myDict:
myDict[myKey] += 1


Dictionaries are very useful to store all kinds of information as Blake T‌ indicated in a comment below:

layers = {'myOutputPoints': 'sourcePoints',
'myOutputLines': 'sourceLines',
'myOutputPolys': 'sourcePolys'}

for out_lyr, in_lyr in layers.items():
# do something with the layer names

Personally, I store my database names and connection strings in dictionaries.

To sort a dictionary by value (not by key) use a lambda function:

for key, val in sorted(dct.items(), key=lambda x: x[1]):
# do something with key and value pair


To create a dictionary from OID and an arbitrary field:

fc = r'C:\path\to\your\featureclass\or\table'
fld_oid = arcpy.Describe(fc).OIDfieldname
fld_other = 'afieldname'
d = {r[0]: r[1] for r in arcpy.da.SearchCursor(fc, (fld_oid, fld_other))}


To conditionally add items to dictionary, use list comprehensions:

d = {r[0]: r[1] for r in arcpy.da.SearchCursor(fc, (fld_oid, fld_other)) if r[1] > somevalue}


Working with lists

Convert string to list and back

myListText = 'A;B;C;1;blah'
myList = myListText.split(';')
myListText =  ';'.join(myList) # convert it back
print myList[0]
# prints 'A'

Convert all items in a list to uppercase

This construct is called a list comprehension and is more efficient than creating a list with a loop.

uppList = [x.upper() for x in lowList]


Compare lists and dictionaries

# Compare two lists and get the items in list 1 but not in list 2
listDifference = list(set(myList1) - set(myList2))

# Do the same, but sort the list on he fly
listDifference = sorted(list(set(myList1) - set(myList2)))

# Compare two lists and get the items that are in both lists
listSame = list(set(myList1) & set(myList2))

# Do the same, but sort the list on he fly
listSame = sorted(list(set(myList1) & set(myList2)))


Combining lists
# suppose you have two lists:
lstOne = ['A','B']
lstTwo = ['C','D','E','F']

# and you want to make a list with the items from both lists, using append
lstOne.append(lstTwo)
# ... will result in lstOne = ['A','B',['C','D','E','F']]
# It adds the list as item 3 (nested list)!

# you can loop through list and add the items:
lstOne = ['A','B']
lstTwo = ['C','D','E','F']
for itemTwo in lstTwo:
lstOne.append(itemTwo)
# ... will result in lstOne = ['A','B','C','D','E','F']

# you can also sum the two lists:
lstOne = ['A','B']
lstTwo = ['C','D','E','F']
lstThree = lstOne + lstTwo
# ... will result in lstThree = ['A','B','C','D','E','F']

# or you can use the 'extend':
lstOne = ['A','B']
lstTwo = ['C','D','E','F']
lstOne.extend(lstTwo)
# ... will result in lstOne = ['A','B','C','D','E','F']


Note that append and extend do not return any value and change the list itself. Using the + operator will create a new list

Enumerate a list

The examples below show what you can achieve by using the enumerate function on a list:

mylist = ["a","b","c","d"]

print list(enumerate(mylist))
# [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]

print [(i, j) for i, j in enumerate(mylist)]
# [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]

print [pair for pair in enumerate(mylist)]
# [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]

# which basically does the same as:
print [(mylist.index(a), a) for a in mylist]
# [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]


An example of using the enumerate function to write a data table to Excel (using the xlwt module) was contributed by Neil Ayres (Thanx Neil):

tbl = "Table to export"
fldOb = arcpy.ListFields(tbl)
FieldNames = [f.name for f in fldOb]
# set up excel output
outXL = tbl + ".xls"
wb = xlwt.Workbook()
SheetNum = 1
# first row in sheet, header line
r = 0
with arcpy.da.SearchCursor(tbl, FieldNames) as cur:
for row in cur:
# if row limit is reached, increment and insert another sheet
if r > 65534:
SheetNum += 1
r = 0
r += 1
[ws.write(r, c, val) for c, val in list(enumerate(row))]
wb.save(outXL)
del wb


Using arcpy.da with lists

Get a list of unique values from a table

import arcpy
set_one = set(r[0] for r in arcpy.da.SearchCursor(FC_or_BL, (fieldName)))



Extract a field value from a table using arcpy.da

def GetFieldValue(FClassOrTable, field):
return arcpy.da.SearchCursor(FClassOrTable, field).next()[0]
def GetMaxFieldValue(FClassOrTable, field):
return sorted(arcpy.da.SearchCursor(FClassOrTable, field), reverse=True)[0][0]
def GetMinFieldValue(FClassOrTable, field):
return sorted(arcpy.da.SearchCursor(FClassOrTable, field))[0][0]


Determine the sum of a column - using arcpy.da and Statistics
def GetSum(FClassOrTable, fldName):
import arcpy
fields = [fldName]
total = 0
with arcpy.da.SearchCursor(FClassOrTable, fields) as cursor:
for row in cursor:
val = row[0]
total += val



If there are many rows, this may not be the fastest way. In that case, you might consider using Statistics_analysis:

import arcpy

def GetSum(FClassOrTable, fldName):
stats = r"C:\temp\Default.gdb\stats" # Point this to a temporary table
arcpy.Statistics_analysis(FClassOrTable, stats, [[fldName, "SUM"]])
row = arcpy.SearchCursor(stats).next()
arcpy.Delete_management(stats)
return row.getValue("SUM_" + fldName)


Arcpy.da cursor tuples and lists considerations

A da search cursor returns a tuple (row), which is immutable. You can, though, convert the tuple to a list, using

tmpList = list(inRow)


If you know the position (index) of the item in the tuple (now a mutable list), you can update it by its index.

tmpList[indexPosition] = valueToAssign


The list can be converted back into a tuple

outRow = tuple(tmpList)


... and use the cursor to insert the tuple as a new row:

inCur.insertRow(outRow)


Using eval in Python

The eval command is very powerful. It allows you to construct a command as text and execute it. Let's have a look at the example below:

import arcpy
import os

dct_conn = {"DEV 10.2": r"C:\Users\xbakker\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\Dev 10.2.sde",
"TST 10.2": r"C:\Users\xbakker\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\Tst 10.2.sde",
"ACC 10.2": r"C:\Users\xbakker\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\Acc 10.2.sde",
"PRO 10.2": r"C:\Users\xbakker\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\Pro 10.2.sde"}

fds = "GEOGENESIS.PROY_INFRAESTRUCTURA"

lst_props = ['canVersion', 'datasetType', 'DSID', 'extent', 'isVersioned',
'MExtent', 'spatialReference.name', 'spatialReference.factoryCode', 'ZExtent']

for amb, con in dct_conn.items():
arcpy.env.workspace = con

lst_fc = arcpy.ListFeatureClasses(feature_dataset=fds)
if lst_fc:
fc = lst_fc[0]

desc = arcpy.Describe(fc)
for prop in lst_props:
try:
print "'{0}'={1}".format(prop, eval("desc.{0}".format(prop)))
except:
pass


In this case, the code loops through 4 connection files and lists the featureclasses stored in a specific feature dataset. For each featureclass it creates a describe object. To list a number of properties (defined in a list called "lst_props") it constructs a string and executes it using the eval command. The magic happens on line 24:

eval("desc.{0}".format(prop))


This will format a string. If the property is "spatialReference.name", the string will be "desc.spatialReference.name". When it is evaluated using eval, it will return the name of the spatial reference. This result is used in another string format function to show:

>> 'spatialReference.name'=Some Spatial Reference Name


Always use functions (when practical)

If you want to know why a function with code is faster than the same code outside a function, read this topic:

http://stackoverflow.com/questions/11241523/why-does-python-code-run-faster-in-a-function

Clean up after  yourself using a finally block

Python scripts that do not delete layers they create create "memory leaks" that can slow down and break ArcGIS!

# temp dataset and layer variables
tmp1, lyr1 = [None]*2
try:
tmp1 = arcpy.CreateScratchName("","", "table")
lyr1 = "tmplyr"
lyr1 = arcpy.MakeFeatureLayer(tmp1, lyr1)
except:
# error handling
finally:
for k in [lyr1, tmp1]:
if k: arcpy.Delete_management(k)


To force the use of a license level you should import the desired license level before importing arcpy. Valid license levels are: 'arcview', 'arceditor' and 'arcinfo' (and 'engine', 'enginegeodb', 'arcserver')

To force a Basic (arcview) license, do this:

import arcview
import arcpy

print "Initial ProductInfo: {0}".format(arcpy.ProductInfo())


Please be aware of this, since you might be claiming two licenses on one machine!

Beware, at 10.3 ArcGIS will consume the highest level available and the import statement of a lower level will not force a lower license level! See thread: import arceditor does not set proper license level

Working with domains

Below a example of obtaining the domain description when reading attributes of a feature. There is no error handling in this snippet, so be aware...

import arcpy, os

def main():
# input data
fgdb = r"D:\Xander\Genesis\Procedimientos\CargarDatos\actualizacion.gdb"
fc_name = "PROY_Proyecto_Gen"
fld_name = "FASE_PROY"

# get featureclass
fc = os.path.join(fgdb, fc_name)

# get field and the domain name
fld = arcpy.ListFields(fc, wild_card=fld_name)[0]
dom_name = fld.domain # returns domain name

# list the domains and get the domain that we need
doms = arcpy.da.ListDomains(fgdb)
dom = GetDomainOnName(dom_name, doms)

# let's loop through the data and return the code and the description
with arcpy.da.SearchCursor(fc, ("OID@", fld_name)) as curs:
for row in curs:
oid = row[0]
code = row[1]
description = GetDescription(code, dom)
print "oid {0}, code={1}, description={2}".format(oid, code, description)

def GetDescription(code, dom):
if dom.domainType == 'CodedValue':
coded_values = dom.codedValues
if code in coded_values:
return coded_values        else:            return "code not found in domain"def GetDomainOnName(name, doms):    dom_found = None    for dom in doms:        if dom.name == name:            dom_found = dom            break    return dom_foundif __name__ == '__main__':    main()  

in my case this yields something like this:

oid 867, code=3, description=Design
oid 868, code=3, description=Design
oid 869, code=9, description=Operation
oid 870, code=9, description=Operation
oid 871, code=9, description=Operation


You will find an even better example at the ArcPy Café written by ArcGIS Team Python:

Get coded-value descriptions when accessing data with cursors | ArcPy Café

Please note that when exporting a featureclass of table, normally the domain code (the actual content of the field) will be exported. To include the domain description in the output, there is a setting to achieve this:

Menu "Geoprocessing", select "Environments...". Next Expand "Fields", and switch on the option "Transfer field domain descriptions".

In code this is done as follows:

import arcpy
from arcpy import env

# settings
outLocation = r"path you output folder"
inFeatures = r"path to input featureclass with code value domains"

# configurar setting
env.transferDomains = True # or "TRANSFER_DOMAINS" is also allowed

# convert to shapefile
outName = "my_output_name.shp"
arcpy.conversion.FeatureClassToFeatureClass(inFeatures, outLocation, outName)


Excellent snippets.

Hi Dan, thanx for the correction!

Regards, Xander

This is great; thanks for sharing!

Thanks...

I did a little organization to but the list stuff together and added a few things. This is a nice little doc Xander, hope we can continue to improve it.

Hi Curtis, thanks for the additions!

This is a great feature. Thank you for starting this.

Very Handy, Thanks!

For formatting leading zeros I still prefer

>>> n = 3

>>> str(n).zfill(3)

'003'

>>>

Or to format anything with preceeding zeros...

>>> "000{}".format(3)

'0003'

>>> "000{}".format( "3" )

'0003'

>>> "000{}".format( "three" )

'000three'

>>>

Been messing with the xlwt module, writing a data table to excel.

During the course of this I discovered the enumerate function.

worksheet write needs a column pointer. This seemed to be an easy way to get it.

An example :

tbl = "Table to export"

fldOb = arcpy.ListFields(tbl)
FieldNames = [f.name for f in fldOb]

# set up excel output
outXL = tbl + ".xls"
wb = xlwt.Workbook()
SheetNum = 1

# first row in sheet, header line
r = 0

with arcpy.da.SearchCursor(tbl, FieldNames) as cur:
for row in cur:
# if row limit is reached, increment and insert another sheet
if r > 65534:

SheetNum += 1
r = 0

r += 1
[ws.write(r, c, val) for c, val in list(enumerate(row))]

wb.save(outXL)
del wb

Can someone format this for me please....

Hi Neil,

Kin regards, Xander

Added an example of using "eval".

Enjoy, Xander

Great code snippets!

One request: I have a lot of tables with fields using coded value domains. Do you have a smart way to retrieve the domain description instead of the code in a row.getValue() request?

Hi Pierter Geert,

I can include some code sample to work with domains in Python, but it would require the use of the data access module. Since you stated "row.getValue()", you may be working with 10.0, the version before the da module was released. Do you have access to the da module?

If so, my recomendation would be to switch to the da cursors, which are much faster. I will add some code sample to retrieve the domain description rather than the code.

Kind regards, Xander

samples with da module are just fine!

Added an example on working with domains.

Pieter Geert van den Beukel‌, next time it might be better to post your question as a thread. This way you can get more valuable answers which are more directed to your specific needs.

Kind regards, Xander

Good work, specially for beginners....

Love it! Thanks for taking the time to put this together.

You're welcome! If you have suggestions to enhance the content or things that may be missing, please let me know...

When working with dictionaries, I found it helpful to iterate the items rather than the generic dictionary so you can name the key and value to be more readable.

import arcpy import os  # Environment variables sourceGDB = r"C:\temp\source\source.gdb" outputGDB = r"C:\temp\output\output.gdb"  # Feature classes dictionary ## Key is destination feature class name (out_lyr) ## Value is source feature class name (in_lyr) layers = {     'myOutputPoints': 'sourcePoints',     'myOutputLines': 'sourceLines',     'myOutputPolys': 'sourcePolys', }  ## End layers  # Export feature classes in dictionary for out_lyr, in_lyr in layers.items():     arcpy.CopyFeatures_management(         os.path.join(sourceGDB, in_lyr),         os.path.join(outputGDB, out_lyr)     )     print "{} copied".format(out_yr)     # write geoprocessing warning messages     if arcpy.GetMessages(1):         print arcpy.GetMessages(1)

I also like to use arcpy.ClearWorkspaceCache_management() to cleanup all my scripts with that connect to SDE.

Thanks This post is very Helpful.

Thanks.

Hi Matthew Russo‌, could you start a thread and post this question in the Python place? You can reference the snippets in the thread: Some Python Snippets , maybe even reference Neil Ayres who provided the original snippet.

This way you will have more possibilities to get a good answer and code syntax highlighting can be applied. (see: Posting Code blocks in the new GeoNet )

I will have a look when you post it.

Kind regards, Xander

This is great information - with tons of detail. Thanks for sharing.

Good snippits.

I would be interested in numbers on whether Statistics_analysis or the 3.x statistics module is faster if the statistics module is supplied a generator that uses da to read a large number of records. Obviously building a list even with a list comprehension and then passing that into a function will have some time costs but so does feeding parameters to the geoprocessor and then reading into your program the output file from the tool.

I know I've seen a few posts where people use a little function to get the "valid" editable fields of a table of feature class. It will at least exclude fields that are not editable (to exclude ObjectID, GlobalID, etc), but I know it did a couple other exclusions. Do you know what I'm referring to? I'd like to see that snippet

field.editable returns a boolean   Field—Help | ArcGIS for Desktop is that what you are referring to?

Partially. I know about the field.editable property, but I'm looking for what properties you check for exclusions if you wanted to dynamically open an update cursor on a table using ListFields(). I know I've seen a few cases where people have posted code with a neat little function that returns the "valid" fields. Sorry, I wish I could explain it better.

field.editable or field.required is pretty well all I can remember and they are set up during creation and can be read...but I can remember if they can be modified once specified...I am leaning to I think not, but I am not on an arc* machine to test.

I think Dan is right. The editable property of the field is the way to go.

def GetEditableFields(fc):

return [fld.name for fld in arcpy.ListFields(fc) if fld.editable]

I tested it with a polygon featureclass with all the supported field types and it returned:

[u'SHAPE', u'FldShort', u'FldLong', u'FldFloat', u'FldDouble', u'FldText', u'FldDate', u'FldBlob', u'FldGuid', u'FldRaster']

It excluded the following fields:

OBJECTID

SHAPE_Length

SHAPE_Area

Here are three examples I found so far. Ironically, I think this snippet from my code is closest to what I was thinking but I still feel like there's something missing.

 # Get list of usable field names (exclude OID and SHAPE fields)fieldNames = [    f.name for f in arcpy.ListFields(inputTable)    if f.type != "Geometry" and f.editable == True]
 Another way of accessing standard fields (not determining the name though) is using tokens in for instance an arcpy.da.SearchCursor. Some examples are:SHAPE@ (A geometry object for the feature.)SHAPE@AREA (A double of the feature's area.)SHAPE@LENGTH (A double of the feature's length.)OID@ (The value of the ObjectID field.)

Re: Spatial join - distance from outline

 def getAllFieldsWithoutGeometryFields(fc):    flds = [fld.name for fld in arcpy.ListFields(fc)]    desc = arcpy.Describe(fc)    fld_length = desc.lengthFieldName    fld_area = desc.areaFieldName    fld_shape = desc.shapeFieldName    if fld_length in flds:        flds.pop(flds.index(fld_length))    if fld_area in flds:        flds.pop(flds.index(fld_area))    if fld_shape in flds:        flds.pop(flds.index(fld_shape))    flds.insert(0, 'SHAPE@')    return flds

so basically you have covered just about every field type possible   someone will still find a way to corrupt  a field

wow, I just realized that the last example of code was written by me and could have been written like:

def getAllFieldsWithoutGeometryFields(fc):

return ['SHAPE@'] + [fld.name for fld in arcpy.ListFields(fc) if fld.editable and fld.type!='Geometry']

You have progressed well young Jedi

Thanks Obi-Dan-Kenobi, it means a lot coming from you

Love it!!!

Ah, found it, thanks Xander Bakker

Been using this sort of syntax to read in features or tables into a dictionary for ages (from code here):

data_dict = {r[0] : r[1:] for r in arcpy.da.SearchCursor(
"DataTbl", [list of columns]}‍‍

But I have been doing a lot of faffing with data ins and outs recently, and got fed up with trying to remember the index numbers for reading the attributes when processing the dictionary.

So, after a bit of messing came up with this :

My input data definition -

# input features
indata_dict = {
"FibreCable" : ["OID@", "Fibre_Type", "Cable_Size", \
"Capacity", "FromDirect", "ToDirectio", \
"InstallDat", "NoUtilised", "SHAPE@"],
"FAT" : ["OID@", "Exchange_C", "MountingTy", "Rack_No", \
"Capacity", "UsedPorts", "FreePorts", "SHAPE@"],
"CableSplice" : ["OID@", "Type", "Joint_No", "Capacity", \
"JB_Pole_Ch", "Cable_ID", "SHAPE@"]
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So I have 3 tables to read.

What I wanted was to be able to access the variables in each row via the column name rather than the index position.

So, it might be obvious to some but not to me...

cable_dict = {r[0] : dict(zip(indata_dict["FibreCable"][1:], r[1:]))
for r in arcpy.da.SearchCursor("FibreCable", indata_dict["FibreCable"])}
‍‍

which results in this sort of content :

>>> cable_dict[1]
{'Cable_Size': u'10.6',
'Capacity': u'48F',
'Fibre_Type': u' ',
'FromDirect': u'GAB01',
'InstallDat': u'2007',
'NoUtilised': 0.0,
'SHAPE@': <Polyline object at 0x2b95cd0[0x2b95f80]>,
'ToDirectio': u'JB48'}‍‍‍‍‍‍‍‍‍

So each row of the dictionary is another dictionary, and when reading this structure to do some processing you can access the contents using the name of the variable rather than its index position.

I suppose I have just replace one problem (remembering the index) with another (remembering the name!!).

Cool trick. I have also used namedtuple with good success in being able to refer to a field name rather than an index.

I'll have a look at that, thanks

real ordered dictionaries are coming to python 3.6... none of this workaround crap to get stuff out of a dictionaries what-ever-I-feel-like at the moment ordering.  Like a real dictionary should have been in the first place

So, blake.terhune how would you use named tuples in a da.SearchCursor?

I don't really think that dictionaries are not ordered is very important.

If you want to read them in a particular order, that is easy to do (as long as the key sorts in the way you want).

Main thing is speed, and they are incredibly fast.

not just sorting, it is FIFO that is nice... especially for unpacking   speed has improved,  Note: in Python 3.6 dict is order-preserving. This virtually eliminates performance concerns.  The release schedule PEP is informative, but the details have to be followed in the links there-in

Most commonly, I actually use it to handle the result of arcpy.ArcSDESQLExecute().

import arcpy
from collections import namedtuple

sde_sdeconn = r"C:\GISConnections\SDE@GTEST.sde"
sql = '''
SELECT LIBRARYPOLYID, NAME, MEET_ROOM_SEAT_CAP
FROM CS.LIBRARY_POLY
'''
row_tpl = namedtuple('row_tpl', 'lid, name, capacity')
cnxn = arcpy.ArcSDESQLExecute(sde_sdeconn)
sqlresult = map(row_tpl._make, cnxn.execute(sql))
for row in sqlresult:
print("{} (ID {}) can seat {}".format(row.name, row.lid, row.capacity))
del cnxn‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

You could use it with an arcpy cursor like this:

import arcpy
from collections import namedtuple
import os

sde_sdeconn = r"C:\GISConnections\SDE@GTEST.sde"
fc = os.path.join(sde_sdeconn, "CS.LIBRARY_POLY")
row_tpl = namedtuple('row_tpl', 'lid, name, capacity')
fields = ["LIBRARYPOLYID", "NAME", "MEET_ROOM_SEAT_CAP"]
cursor_result = map(row_tpl._make, [row for row in arcpy.da.SearchCursor(fc, fields)])
for row in cursor_result:
print("{} (ID {}) can seat {}".format(row.name, row.lid, row.capacity))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Thanks Blake for the tips on named tuples....

Version history
Last update:
‎12-12-2021 03:37 AM
Updated by: