I'll go with the append.
#UNTESTED! lutTbl = r"C:\temp\test.gdb\lookuptable" mainTbl = "r"C:\temp\test.gdb\maintable" lutDict= dict([(r[0], (r[1], r[2])) for r in arcpy.da.SearchCursor(lutTbl, ["NAME","ADDRESS","CITY")]) arcpy.AddField_managment(mainTbl, "ADDRESS", "TEXT", "", "", "75") arcpy.AddField_managment(mainTbl, "CITY", "TEXT", "", "", "30") updateRows = arcpy.da.UpdateCursor(mainTbl, ["NAME","ADDRESS","CITY"]) for updateRow in updateRows: nameValue = updateRow[0] if nameValue in lutDict: updateRow[1] = lutDict[nameValue][0] #Address updateRow[2] = lutDict[nameValue][1] #City else: print "Could not locate address/city info for " + str(nameValue) updateRows.updateRow(updateRow) del updateRow, updateRows
# fix date, returns a dictionary with sortable values def FixDate(in_date): month_dict = {'Jan':'01','Feb':'02','Mar':'03','Apr':'04', 'May':'05','Jun':'06','Jul':'07','Aug':'08', 'Sep':'09','Oct':'10','Nov':'11','Dec':'12'} # split date string by space and remove comma in_date_items = in_date.replace(',','').split() # separate year,month,day year = in_date_items[-1] month = month_dict[in_date_items[0]] day = in_date_items[1].zfill(2) # sort date dictionary sort_dict = {} sort_dict[in_date] = '/'.join([year,month,day]) return sort_dict # your dictionary of applicants apps = {'Smith, John':[['Smith, John', 'Jan 10, 2013', 'Applied'], ['Smith, John', 'Feb 3, 2013' 'Assigned'], ['Smith, John', 'Mar 25, 2013', 'Tested'], ['Smith, John', 'Jun 11, 2013', 'Hired']], 'Smith, Jack':[['Smith, Jack', 'Jan 10, 2013', 'Applied'], ['Smith, Jack', 'Feb 7, 2013', 'Assigned'], ['Smith, Jack', 'Mar 25, 2013', 'Tested'], ['Smith, Jack', 'Jun 5, 2013', 'Rejected']], 'Smith, Kim':[['Smith, Kim', 'Jan 10, 2013', 'Applied'], ['Smith, Kim', 'Feb 12, 2013', 'Rejected']]} # return most recent action for each person action_dict = {} for name, info in apps.iteritems(): #iterates through each list of actions to find most recent action for details in info: dates = {} dates[details[1]] = details[-1] recent = sorted(list(FixDate(dt) for dt in dates.keys()))[-1] # grabs the last item (max in this case) action_dict[name] = [recent.keys()[0], dates[recent.keys()[0]]] # copies this into a new dictionary print name, recent.keys()[0], dates[recent.keys()[0]]
print action_date['Smith, John']
for name, action in action_dict.iteritems(): print name, action
import arcpy dbf = r'C:\TEMP\Test_Table.dbf' # Get Unique names fields = [f.name for f in arcpy.ListFields(dbf) if f.type != 'OID'] with arcpy.da.SearchCursor(dbf, fields) as rows: table = list(r for r in rows) # table as tuple names = list(set(r[0] for r in table)) # New dictionary people_dict = {} # Make each unique name the key # return all matches as list for values for name in names: people_dict[name] = [r for r in table if r[0] == name] # print matching records for each name for name,matches in people_dict.iteritems(): print name,matches print '\n\n'
Alex Olsen [(u'Alex Olsen', u'516 N Main St', u'West Branch', 61500.0)] Shelly Fields [(u'Shelly Fields', u'618 N Ward St', u'Macomb', 53500.0)] John Smith [(u'John Smith', u'1321 400th St', u'Tipton', 54000.0), (u'John Smith', u'222 E Third St', u'Lisbon', 47800.0), (u'John Smith', u'212 S 1st St', u'Tipton', 80000.0)] Steve Johnson [(u'Steve Johnson', u'447 S Fisk St', u'Macomb', 43400.0), (u'Steve Johnson', u'214 S 1st St', u'Tipton', 72000.0)]
# print all address, city for each match to name for name,matches in people_dict.iteritems(): print name, [(mat[1],mat[2]) for mat in matches] print '\n\n'
# print address,city for John Smith for name,attributes in people_dict.iteritems(): if name == 'John Smith': for att in attributes: print att[1:3]
# Get sum of income for each name for name,matches in people_dict.iteritems(): print name, sum(mat[-1] for mat in matches)
compKeyDict = {('smith, john', 1): "cat street", ('smith, john', 2): "dog street", ('smith, jack', 1): "clown street"} johnSmithKeysList = [key for key in compKeyDict if "smith, john" == key[0]]
can the composite key approach involve a list of items, or is it limited to single values and tuples?
testList = [1,2,3,4] testTuple = tuple(list)
You have to get creative - I routinely deal with one to many or many to manys by using either composite keys and/or composite look up values. A basic example:compKeyDict = {('smith, john', 1): "cat street", ('smith, john', 2): "dog street"} compValDict = {'smith, john': ["cat street", "dog street"]}
compKeyDict = {('smith, john', 1): "cat street", ('smith, john', 2): "dog street"} compValDict = {'smith, john': ["cat street", "dog street"]}
To answer some of your questions:Yes. But you can use tuples as keys as well (aka a composite key) - which of course also have to be unique. You have to get creative - I routinely deal with one to many or many to manys by using either composite keys and/or composite look up values. A basic example:compKeyDict = {('smith, john', 1): "cat street", ('smith, john', 2): "dog street"} compValDict = {'smith, john': ["cat street", "dog street"]}I would put a wager in that anything (well pretty much anything) you can do in a RDBMS you can also do much faster and cheaper using dictionaries. All it takes is imagination and a lot of conditional expressions!If it helps, here's a practical example of making use of a composite value dictionary where the sorted order of the many values are important to the overal analysis: http://forums.arcgis.com/threads/89835-brainteaser-viewshed-wind-turbines-the-more-you-see-the-worse-it-gets...?p=320549&viewfull=1#post320549
Does the key value have to be unique?
...what happens if there are two people named John Smith that have different addresses in the look up table?
I'll go with the append.Don't!Going the Python route (reading the join tables(s) into a dictionary using a search cursor and then updating the main table via an update cursor is by far the fastest method. This is true in v10.0 and below, but is especially true in v10.1+ using ethe data access cursors. In addition to faster processing, this method is far more flexible in that allows for all sorts of error handeling and whatnot through conditional expressions.For example, say you want to get the fields "ADDRESS" and "CITY" into the main table (key field being "NAME"):lutTbl = r"C:\temp\test.gdb\lookuptable" mainTbl = "r"C:\temp\test.gdb\maintable" lutDict= dict([(r[0], (r[1], r[2])) for r in arcpy.da.SearchCursor(lutTbl, ["NAME","ADDRESS","CITY"])]) arcpy.AddField_managment(mainTbl, "ADDRESS", "TEXT", "", "", "75") arcpy.AddField_managment(mainTbl, "CITY", "TEXT", "", "", "30") updateRows = arcpy.da.UpdateCursor(mainTbl, ["NAME","ADDRESS","CITY"]) for updateRow in updateRows: nameValue = updateRow[0] if nameValue in lutDict: updateRow[1] = lutDict[nameValue][0] #Address updateRow[2] = lutDict[nameValue][1] #City else: print "Could not locate address/city info for " + str(nameValue) updateRows.updateRow(updateRow) del updateRow, updateRows
lutTbl = r"C:\temp\test.gdb\lookuptable" mainTbl = "r"C:\temp\test.gdb\maintable" lutDict= dict([(r[0], (r[1], r[2])) for r in arcpy.da.SearchCursor(lutTbl, ["NAME","ADDRESS","CITY"])]) arcpy.AddField_managment(mainTbl, "ADDRESS", "TEXT", "", "", "75") arcpy.AddField_managment(mainTbl, "CITY", "TEXT", "", "", "30") updateRows = arcpy.da.UpdateCursor(mainTbl, ["NAME","ADDRESS","CITY"]) for updateRow in updateRows: nameValue = updateRow[0] if nameValue in lutDict: updateRow[1] = lutDict[nameValue][0] #Address updateRow[2] = lutDict[nameValue][1] #City else: print "Could not locate address/city info for " + str(nameValue) updateRows.updateRow(updateRow) del updateRow, updateRows
Anyway - dictionaries are great. I do all my fancy SQL-like stuff in them now... especially now with 64-bit arcpy. Data tables have kazillions records? No problem and crazy fast!!! No need for fancy RDBMs...
I'll go with the append.Don't!Going the Python route (reading the join tables(s) into a dictionary using a search cursor and then updating the main table via an update cursor is by far the fastest method. This is true in v10.0 and below, but is especially true in v10.1+ using ethe data access cursors. In addition to faster processing, this method is far more flexible in that allows for all sorts of error handeling and whatnot through conditional expressions.For example, say you want to get the fields "ADDRESS" and "CITY" into the main table (key field being "NAME"):#UNTESTED! lutTbl = r"C:\temp\test.gdb\lookuptable" mainTbl = "r"C:\temp\test.gdb\maintable" lutDict= dict([(r[0], (r[1], r[2])) for r in arcpy.da.SearchCursor(lutTbl, ["NAME","ADDRESS","CITY")]) arcpy.AddField_managment(mainTbl, "ADDRESS", "TEXT", "", "", "75") arcpy.AddField_managment(mainTbl, "CITY", "TEXT", "", "", "30") updateRows = arcpy.da.UpdateCursor(mainTbl, ["NAME","ADDRESS","CITY"]) for updateRow in updateRows: nameValue = updateRow[0] if nameValue in lutDict: updateRow[1] = lutDict[nameValue][1] #Address updateRow[2] = lutDict[nameValue][2] #City else: print "Could not locate address/city info for " + str(nameValue) updateRows.updateRow(updateRow) del updateRow, updateRows
#UNTESTED! lutTbl = r"C:\temp\test.gdb\lookuptable" mainTbl = "r"C:\temp\test.gdb\maintable" lutDict= dict([(r[0], (r[1], r[2])) for r in arcpy.da.SearchCursor(lutTbl, ["NAME","ADDRESS","CITY")]) arcpy.AddField_managment(mainTbl, "ADDRESS", "TEXT", "", "", "75") arcpy.AddField_managment(mainTbl, "CITY", "TEXT", "", "", "30") updateRows = arcpy.da.UpdateCursor(mainTbl, ["NAME","ADDRESS","CITY"]) for updateRow in updateRows: nameValue = updateRow[0] if nameValue in lutDict: updateRow[1] = lutDict[nameValue][1] #Address updateRow[2] = lutDict[nameValue][2] #City else: print "Could not locate address/city info for " + str(nameValue) updateRows.updateRow(updateRow) del updateRow, updateRows
updateRow[1] = lutDict[nameValue][1]
row[1] = PADataDict[PKValue][0]
Assuming the join fields are indexed on all of the tables, option 1 blew away cursors under the pre-da version cursors. Have not tested the da version cursors. However, even ArcObjects cursors perform slower than joins, so I doubt Option 2 would work faster even with a da cursor. Hitting tables with repeated queries is much slower than a join, which does just one correlation of the tables, because each new query has to reset the table read, as far as I know. Main reason to not use joins would be if there was a 1:M or M:M relationship to traverse.
Ok I have a feature class (260k records) and 3 tables. I need to get specific information (about 20 fields) from each into a new feature class. I have several different ways to do this and I�??m curious if anyone has done any performance testing and knows which of my options would be �??best�?�.Option 1I could create several joins and append the data into the new feature class, with some additional field calculations for cleanup.Option 2Create a python script that would loop through each record, search cursor the other tables for data, then insert cursor into the new feature class.Thank youAlan
Signed in members can post, follow updates, and more. New here? Register a free account.
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.