Latest Contributions by ModernElectric

Caleb-      It appears the script is and is not working.... When I run the script in Python IDLE - it shows that it is running and I am not getting errors back... But when I go look at the feature class - the fields are not updated. They are still showing the same data that I had before I ran the script. So it tells me - the script is not updating the field attributes. I can make it work with only (1) field - but when I add additional fields to the list - that is when it does not update. Any ideas?? ... View more

Chris: Caleb's script is far more advanced than the script I was working on originally with you.  At this point the only help I can provide you with is on the original script as I do not have the time to digest and understand Caleb's script. Michael- Not a problem....I am working on doing my homework on your last post dealing with the index(s) ... View more

Michael & Caleb-    I thought I had the code(s) good to go - but apparently I was wrong and starting to get a little frustrated because I do not understand what I am doing wrong. I can get the code to work just fine BUT with only (1) field at a time. When I add multiple fields to my list in both codes - it does not update anything. Here is the MemoryTableTools.py code: mport arcpy from arcpy import env from MemoryTableTools import * #Set environment settings env.workspace = "C:\\MEWCo GIS System\\Electric System\\MEWCo_Electric_Model-LOCAL.gdb" # Set Local variables inFeatures = "SERVICE_LOCATION" joinField = "POLE_NUMBER" joinField2 = "MAPNO" joinTable = "CIS_Account_Data" up_fields = ['CUSTOMER_NAME', 'ACCOUNT_NUMBER', 'TWACS_NUMBER', 'ACCOUNT_STATUS'] jn_fields = ['CUSTNAME', 'ACCOUNTNO', 'METERNO', 'ACCOUNT_STATUS_DESC'] # Run Attribute Update MemoryTableTools.AttributeUpdate(inFeatures, 'POLE_NUMBER', up_fields, joinTable,'MAPNO',jn_fields) Here is the other Python script that I am using to run the update: import sys, traceback, arcpy, os from os import path as p def AttributeUpdate(source_table, in_field, update_fields, join_table, join_key, join_values):     try:         """ Updates fields from one table to user defined fields in another table"""         # Check input parameters         if not arcpy.Exists(source_table):             print source_table + ' not found!\nPlease verify that full path of table exists'             sys.exit()         if not arcpy.Exists(join_table):             print join_table + ' not found!\nPlease verify that full path of table exists'             sys.exit()         if in_field not in [f.name for f in arcpy.ListFields(source_table)]:             print in_field + ' not found in ' + p.basename(source_table)             sys.exit()         if join_key not in [f.name for f in arcpy.ListFields(join_table)]:             print join_key + ' not found in ' + p.basename(join_table)             sys.exit()         for fld in update_fields:             if fld not in [f.name for f in arcpy.ListFields(source_table)]:                 print fld + ' not found in ' + p.basename(source_table)                 print 'Please verify that field names match in ' + p.basename(source_table)                 sys.exit()         for fldb in join_values:             if fldb not in [f.name for f in arcpy.ListFields(join_table)]:                 print fldb + ' not found in ' + p.basename(join_table)                 print 'Please verify that field names match in ' + p.basename(join_table)                 sys.exit()         if not type(join_values) == list:             join_values = list(join_values)         if not type(update_fields) == list:             update_fields = list(update_fields)         # Make sure there is matching number of join and update fields         update_dict = {}         if len(update_fields) == len(join_values):             for i in range(len(update_fields)):                 update_dict[join_values] = update_fields                             for k,v in update_dict.iteritems():                 # Create Dictionary                 path_dict = {}                 srows = arcpy.SearchCursor(join_table)                 for srow in srows:                     keyrow = srow.getValue(join_key)                     valrow = srow.getValue(k)                     path_dict[keyrow] = valrow                                 # Update Cursor                 urows = arcpy.UpdateCursor(source_table)                 for row in urows:                     upkey = row.getValue(in_field)                     try:                         if upkey in path_dict:                             row.setValue(v, path_dict[upkey])                             urows.updateRow(row)                     except:                         pass  # skip nulls                 print '\'%s\' field in "%s" updated successfully' %(v, p.basename(source_table))             del row, urows, srow, srows                   else:             print 'ERROR:  Number of update fields and value fields does not match'                     except:         print '\nERROR in AttributeUpdate Function.\nPlease Check input parameters and try again.' if __name__ == '__main__':     shp = r'C:\MEWCo GIS System\Electric System\MEWCo_Electric_Model-LOCAL.gdb\Electric\SERVICE_LOCATION'     dbf = r'C:\MEWCo GIS System\Electric System\MEWCo_Electric_Model-LOCAL.gdb\CIS_Account_Data'     # Attribute Update     up_fields = ['CUSTOMER_NAME', 'ACCOUNT_NUMBER', 'TWACS_NUMBER', 'ACCOUNT_STATUS']     jn_fields = ['CUSTNAME', 'ACCOUNTNO', 'METERNO', 'ACCOUNT_STATUS_DESC']     AttributeUpdate(shp,'POLE_NUMBER',up_fields,dbf,'MAPNO',jn_fields) Can you take a quick look and see what I am doing wrong - why I can get it to work with (1) field but when I add additional fields to it - it will not work at all. Thank you ... View more

Michael- Thank you for the update. I am experimenting with Caleb's script to see if I can get it to work properly. I might do some research on the arcpy.AddIndex tool and rework my original script with the Join and Calculate Field tools to see what works best for our environment. Thanks again Chris and Caleb: After further research I think I have found out why my script which performs the same action on a much larger dataset runs quickly.  I create an index on the field in the table that I am using to join to the feature class before I create the join.  I took the model from another person who built the model in modelbuilder and I exported it to a python script where I made some tweaks (That is why I was unaware of the index previously). As such Chris, you could also add a arcpy.AddIndex_management step to your process in the CIS table which should dramatically speed up your original script (Only if you can not get Caleb's more advanced script to work for you). ... View more

Caleb-   Yes - some of that makes sense (Use of Dictionaries)... I plan on doing some more research to see if I can use this with other things/ideas with my other geodatabases. I work with electric and water utility infrastructures and been trying to develop some scripts/processes so I can integrate other parts of our company to GIS (CIS data, SCADA, TWACs, ETC). Again..thank you very much.... This is going to cut down ALOT of time in updating my feature classes with tables from our CIS Oracle database Chris @ Chris: I am glad you were able to get it to work.  As for having it do more fields you just need to make a list of fields to updated in table A and a list of the fields from table B that contain the values.  However, you must make sure to keep the fields in the correct order for each list. For example: list_A = ['field_1', 'field_2', field_3', etc...]  # for table A list_B = ['field_A', 'field_B', field_C', etc...]  # for table B I have never tested this on more than 7 or so fields at a time but I am sure that it will work the same for 20. Now I will try to explain the dictionary usage: Most of my script is error handling, but the main meat of the script that is actually updating the table is this part:                 # Create Dictionary                 path_dict = {}                 srows = arcpy.SearchCursor(join_table)                 for srow in srows:                     keyrow = srow.getValue(join_key)                     valrow = srow.getValue(k)                     path_dict[keyrow] = valrow                                 # Update Cursor                 urows = arcpy.UpdateCursor(source_table)                 for row in urows:                     upkey = row.getValue(in_field)                     try:                         if upkey in path_dict:                             row.setValue(v, path_dict[upkey])                             urows.updateRow(row)                     except:                         pass  # skip nulls So a dictionary is just made up of pairs consisting of keys and values like this { key : value}.  You can look up a value by supplying the key...Values can be any value (even objects and entire lists).  The above portion uses a search cursor on table B (with the values) to make a dictionary using each record in the join field (your common field) as a key.  The corresponding value will be one of the fields from the "join_fields" parameter.  It creates a separate dictionary for each one. Next, an update cursor is used to search through the table and if the value in the join_field in table A is a key in the dictionary, it will update the corresponding "update_field" for that row.  I have an exception set up so that it will skip nulls when working through shapefiles.  I hope this makes sense. @ Michael I had not tested the field calculator in arcpy land since I upgraded to 10.1.  However, after a quick test it seems that it runs much more efficiently now than it did in the past.  I have since gotten a new computer so maybe that is why it was so slow in the past.  I am glad you find this useful.  I have another function from the same script that copies fields from one table to another with the same schema so you can avoid the "field mapping" type operation like in the AttributeUpdate().  I can post that if anyone is interested. ... View more

Caleb and Michael-    I got it all to work. The script(s) with the dictionaries did EXACTLY what I wanted it to so with almost NO time at all. Now its just going to take me awhile to program the fields into the script and run an overnight test...... Next step - reading up on dictionaries and indexes so I can attempt to understand this large script that Caleb sent me.. Thank you all (Caleb & Michael) very much for all your help and input.. Think my blood pressure will come down now too... ... View more

Caleb-    I got your process/script to WORK.......It took about 15 seconds for the MEMBER_NUMBER field to update from the CIS_Account_data table.... Now - I know I have alot of work to do since I have about 20 fields in my electric model and about 15 in my water model to have this done too... What would be my next step so this works for all of my fields Example: Customer name:     Feature Class - CUSTOMER_NAME     Table - CUSTNAME Thank you very much!!!!! ... View more

Caleb/Michael- All is correct. What I am doing is simple ArcGIS Python scripting 101.... When it comes to the talk of dictionaries...indexes...etc (Advanced Python scripting) I am lost...... There has to be an easier way for a rookie python scripter like myself to be able to program a process that takes Field "Member Number" from a file Geodatabase table and copy it to a field in a file geodatabase "member number" after it makes a join. Would it be easier:    Add Join (Table to Feature Class) - using Map Number/MAPNO as the join attribute    Export to a new Feature Class    Calculate Field(s)    Delete fields that use to be the table    Rename feature class back to the original name Is that overkill or would to work??? Caleb: Your solution is very elegant compared to the script that Chris wrote and I helped him modify.  I cannot speak for Chris, but your script is advanced and might be difficult for a person new to python scripting. It just seems that Chris' script has a minor issue that can be quickly corrected (once its found) as he is working with such a small dataset. ... View more

Michael- I went to my Feature Class and Table just to verify. The (2) fields I am using to Join the table to the feature class are the same. Also the (2) fields (Member Number) are both [TEXT] field type. Caleb: Is there any reason why Chris' original script should take so long to run? I am working with a much larger feature class and table performing the same type of join and calculate geoprocess and it completes in 5 minutes. Is there any chance it could be due to field type differences between the join field in the table and the field that is being calculated in the feature class? ... View more

Caleb- Went through and did exactly what you said over again with the script you included in the latest E-Mail....And I get this: ImportError: Bad magic number in C:/Python27/ArcGIS10.1/Lib\weakref.pyc Also - when I open the script in Python IDLE - it says version: Python 2.6.5 - is this a big deal or do I need to upgrade? Thanks Hmm, that is strange.  Did you save the script as MemoryTableTools.py and put it in your Python folder?  I guess you must have since you did not get an import error.  Perhaps you could try the namespace then: import arcpy from arcpy import env import MemoryTableTools # Set environment settings env.workspace = "C:\\MEWCo GIS System\\Electric System\\MEWCo_Electric_Model-LOCAL.gdb" # Set Local variables: inFeatures = "SERVICE_LOCATION"                         joinField = "POLE_NUMBER" joinField2 = "MAPNO" joinTable = "CIS_Account_Data" up_fields = ['MEMBER_NUMBER'] jn_fields = ['MEMBERNO'] # Run Attribute Update MemoryTableTools.AttributeUpdate(inFeatures, 'POLE_NUMBER', up_fields, joinTable,'MAPNO',jn_fields) ... View more

Caleb- Its going to take me awhile to figure out the dictionaries and all.... This new script you have me - I did exactly as you directed and get an error message: NameError: name 'AttributeUpdate' is not defined Chris So for your case if you save the script as MemoryTableTools.py like I have, this code should work: import arcpy from arcpy import env from MemoryTableTools import * # Set environment settings env.workspace = "C:\\MEWCo GIS System\\Electric System\\MEWCo_Electric_Model-LOCAL.gdb" # Set Local variables: inFeatures = "SERVICE_LOCATION"                         joinField = "POLE_NUMBER" joinField2 = "MAPNO" joinTable = "CIS_Account_Data" up_fields = ['MEMBER_NUMBER'] jn_fields = ['MEMBERNO'] # Run Attribute Update AttributeUpdate(inFeatures, 'POLE_NUMBER', up_fields, joinTable,'MAPNO',jn_fields)         Keep in mind that this works on multiple fields at a time if you need to, the above example just works on one field from your first post.  Also, be sure to save the script in a path that can be found by python (PYTHONPATH).  A directory like this: C:\Python27\ArcGIS10.1\Lib ... View more

Chris: When is the feature class being saved to disk, as I do not see this in your code sample? arcpy.CalculateField_management(layerName, "SERVICE_LOCATION.MEMBER_NUMBER", '!MEMBERNO!', "PYTHON") This appears to be incorrect as layerName should be the in_table expression should be !CIS_Account_Data.MEMBERNO! as you need to explicitly specify that this is coming from a join field. No Dice.... I am working with just this (1) field and after waiting 20 minutes - it does not complete. Is is possible this temp feature layer is causing issues?? ... View more

Hi, I had the same problem with AddJoin + CalculateField efficiency. One solution could be creating indexes on fields you're using in join if you hadn't done this before. If this won't help (sometimes it's not enough) I would suggest using combination of arcpy.sa.SearchCursor and python dictionary to collect right data from table, and then arcpy.da.UpdateCursor to update featureclass fields. It take some time to write, but it's very efficient. If you need some details feel free to ask. Best Regards. Arek Arek- Thank you for the reply. Creating the Feature Layer and doing the Join seems to work just fine without any hold-up or lagging.... This is being done outside of ArcMap outside of an Edit Session... My intention is to have this done automatically at night and the only way I have figured out to do it is to have a Python Script and run a schedule task using the Windows Task Scheduler. Forgive me - but I am still really new to Python and not sure how to use or write indexes and python dictionaries......I have done some (very little reading on it)..... Any ideas???? Thank you ... View more

Here is the current updated Python script that I have. I tried to run the script through Python IDLE and just running it by double-clicking on the file which brings up DOS... and it still seems to lag and lag. I will post this on the Python form and see if someone else can think of something. ## Set the necessary product code import logging # Import arcpy module import arcpy from arcpy import env # if run in ArcGIS Desktop, show messages, also print to log def log(method, msg):     print msg     method(msg) logging.basicConfig(         level=logging.INFO,         format='%(asctime)s %(levelname)-s %(message)s',         datefmt='%a, %d %b %Y %H:%M:%S',         filename='ServiceLocation_CISAccountData_Updates.log',         filemode='w'         ) log(logging.info, "Updating Service Locations Feature Class Begins") # Set environment settings env.workspace = "C:\\MEWCo GIS System\\Electric System\\MEWCo_Electric_Model-LOCAL.gdb" # Set Local variables: inFeatures = "SERVICE_LOCATION"                         # Service Location Feature Class layerName = "SERVICE_LOCATION_LAYER"                    # Service Location Feature Layer fieldName1 = "SERVICE_ADDRESS" expression1 = "CIS_Account_Data.SERVICE_ADDRESS" joinField = "POLE_NUMBER" joinField2 = "MAPNO" joinTable = "CIS_Account_Data" # Create a Feature Layer from Service_Location Feature Class log(logging.info, "Create Feature Layer for Service Location Feature Class") arcpy.MakeFeatureLayer_management(inFeatures, layerName) # Join the feature layer to a table log(logging.info, "Join Table to Feature Layer Service Location") arcpy.AddJoin_management(layerName, joinField, joinTable, joinField2)     # Process: Calculate Field (Member Number) log(logging.info, "Update Member Number Field") arcpy.CalculateField_management(layerName, "SERVICE_LOCATION.MEMBER_NUMBER", '!MEMBERNO!', "PYTHON") log(logging.info, "Complete") ... View more