Updating Feature Class Attributes from Table

5010
19
04-08-2013 08:03 AM
ModernElectric
Frequent Contributor
Greetings-

     I have tried multiple things and trying to find a better/easier/faster way of doing this:

I have a File Geodatabase Feature Class (Service_Locations) that has around 9500 records in it. This is the location and data for our Electric meters. The data behind it (Customer info, meter attributes, ETC) are stored/maintained/updated in our CIS system (Data held in an Oracle Database). Everynight - I have a python script that automatically exports a View from the Oracle Database (Customer data, Meter Data) and brings it directly into the File Geodatabase as a Geodatabase Table.

What I have done in the past is to:
    Add Join (Feature Class Table TO Service_Locations FC)
    Calculate Field tool to say - Bring this field from the table (Meter Number) into this field of the FC (TWACs Number).
I have done this inside and edit session and a data model to Python script (Outside of an edit session)
Each way is very time consuming and does not work fast.

QUESTION: Are there different tools that I can use to make this process much faster so I can build a Python Script to run automatically at night? The Join and Calculate Field tools are not working for me anymore..

Thank you
0 Kudos
19 Replies
ModernElectric
Frequent Contributor
I'm curious on one thing - is doing the "Make Feature Layer" and "Add Join" have anything to do with the long time it takes to do the
"Calculate Field" tool outside of an edit session??


Chris - Can you just run the "Make Feature Layer" and "Add Join" part of your python script and see how long it takes? This could determine that this is the cause of the long processing time? I would even just run the "Make Feature Layer" separately from the "Add Join" to determine which process is causing the long processing time.


I went through and just did the "Make Feature Layer" and "Add Join" tool on its own and it was done within 10 seconds. For some
reason - the calculate field tool takes a long time??? Not sure how to speed it up or if there is another tool that works the same way??

Is it a problem that I am trying to "Override" (E) values/data inside the field? Would it make any difference if the field was blank??
0 Kudos
MichaelVolz
Esteemed Contributor
arcpy.CalculateField_management(SERVICE_LOCATION_FL__2_, "SERVICE_LOCATION.MEMBER_NUMBER", "[CIS_Account_Data.MEMBERNO]", "VB", "")


Chris:

Maybe because you are using the VB syntax it is taking a long time for the CalculateField_management process.  Try converting this to PYTHON with the appropriate syntax for field names [field name] as per the documentation.

http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//00170000004m000000
0 Kudos
ModernElectric
Frequent Contributor
arcpy.CalculateField_management(SERVICE_LOCATION_FL__2_, "SERVICE_LOCATION.MEMBER_NUMBER", "[CIS_Account_Data.MEMBERNO]", "VB", "")


Chris:

Maybe because you are using the VB syntax it is taking a long time for the CalculateField_management process.  Try converting this to PYTHON with the appropriate syntax for field names [field name] as per the documentation.

http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//00170000004m000000


How exactly would I write that? I am still trying to figure out Python
0 Kudos
MichaelVolz
Esteemed Contributor
Chris:

Here is a code snippet where I use python syntax instead of VB syntax:

arcpy.CalculateField_management(rdcl_Layer, "MUNI_NUM_RIGHT_UNPAD", '!MUNI_NUM_RIGHT!.lstrip("0")', "PYTHON", "")


For your code:  arcpy.CalculateField_management(SERVICE_LOCATION_FL__2_, "SERVICE_LOCATION.MEMBER_NUMBER", "[CIS_Account_Data.MEMBERNO]", "VB", "")

change it to

arcpy.CalculateField_management(SERVICE_LOCATION_FL__2_, "SERVICE_LOCATION.MEMBER_NUMBER", '!CIS_Account_Data.MEMBERNO!', "PYTHON", "")

I would even set the env variable as the destination file geodatabase so you do not need to reference it in your call to CalculateField_management

Near the top of your script below import arcpy

from arcpy import env
env.workspace = "C:\\MEWCo GIS System\\Electric System\\MEWCo_Electric_Model-LOCAL.gdb\\CIS_Account_Data"

then your code could be

arcpy.CalculateField_management(SERVICE_LOCATION_FL__2_, "SERVICE_LOCATION.MEMBER_NUMBER", '!MEMBERNO!', "PYTHON", "")

I'm not really sure this is the issue as I use VB notation in a CalculateField_management process where I have 200,000 records and it only takes 5 minutes as opposed to hours.
0 Kudos
ModernElectric
Frequent Contributor
Chris:

Here is a code snippet where I use python syntax instead of VB syntax:

arcpy.CalculateField_management(rdcl_Layer, "MUNI_NUM_RIGHT_UNPAD", '!MUNI_NUM_RIGHT!.lstrip("0")', "PYTHON", "")


For your code:  arcpy.CalculateField_management(SERVICE_LOCATION_FL__2_, "SERVICE_LOCATION.MEMBER_NUMBER", "[CIS_Account_Data.MEMBERNO]", "VB", "")

change it to

arcpy.CalculateField_management(SERVICE_LOCATION_FL__2_, "SERVICE_LOCATION.MEMBER_NUMBER", '!CIS_Account_Data.MEMBERNO!', "PYTHON", "")

I would even set the env variable as the destination file geodatabase so you do not need to reference it in your call to CalculateField_management

Near the top of your script below import arcpy

from arcpy import env
env.workspace = "C:\\MEWCo GIS System\\Electric System\\MEWCo_Electric_Model-LOCAL.gdb\\CIS_Account_Data"

then your code could be

arcpy.CalculateField_management(SERVICE_LOCATION_FL__2_, "SERVICE_LOCATION.MEMBER_NUMBER", '!MEMBERNO!', "PYTHON", "")

I'm not really sure this is the issue as I use VB notation in a CalculateField_management process where I have 200,000 records and it only takes 5 minutes as opposed to hours.


Michael-

That helps out alot...I am not getting any error messages so I wrote the code correctly with your help. But that still does not solve my problem. It seems to be freezing or not finishing when I run the calculate field tool. The Geodatabase and all of the data is on my Local drive instead of being ran over the network. Also - I am running the Python script with Python IDLE 2.6.5. Do you think that might be a reason why it is taking so long to run?? You have been a great help in working with me on the code. I am not sure if
its the Geodatabase or the version of Python I am using or something else. I am only updating 9000 records and working with (1) field right now. Anything else you can possibly think of that I am doing wrong???
0 Kudos
MichaelVolz
Esteemed Contributor
Two things:

I would post this thread to the Python subforum as you might can help from more advanced python users.

Can you provide the remainder of your code as I'm wondering if the bottleneck might be with saving the feature class to disk as the arcpy.MakeFeatureLayer_management should only save the feature class to memory so you still need to save the feature class permanently to your file gdb?
0 Kudos
ModernElectric
Frequent Contributor
Here is the current updated Python script that I have. I tried to run the script through Python IDLE and just running it by double-clicking on the file which brings up DOS... and it still seems to lag and lag.

I will post this on the Python form and see if someone else can think of something.

## Set the necessary product code
import logging

# Import arcpy module
import arcpy
from arcpy import env


# if run in ArcGIS Desktop, show messages, also print to log
def log(method, msg):
    print msg
    method(msg)

logging.basicConfig(
        level=logging.INFO,
        format='%(asctime)s %(levelname)-s %(message)s',
        datefmt='%a, %d %b %Y %H:%M:%S',
        filename='ServiceLocation_CISAccountData_Updates.log',
        filemode='w'
        )

log(logging.info, "Updating Service Locations Feature Class Begins")

# Set environment settings
env.workspace = "C:\\MEWCo GIS System\\Electric System\\MEWCo_Electric_Model-LOCAL.gdb"

# Set Local variables:
inFeatures = "SERVICE_LOCATION"                         # Service Location Feature Class
layerName = "SERVICE_LOCATION_LAYER"                    # Service Location Feature Layer
fieldName1 = "SERVICE_ADDRESS"
expression1 = "CIS_Account_Data.SERVICE_ADDRESS"
joinField = "POLE_NUMBER"
joinField2 = "MAPNO"
joinTable = "CIS_Account_Data"

# Create a Feature Layer from Service_Location Feature Class
log(logging.info, "Create Feature Layer for Service Location Feature Class")
arcpy.MakeFeatureLayer_management(inFeatures, layerName)

# Join the feature layer to a table
log(logging.info, "Join Table to Feature Layer Service Location")
arcpy.AddJoin_management(layerName, joinField, joinTable, joinField2)
    
# Process: Calculate Field (Member Number)
log(logging.info, "Update Member Number Field")
arcpy.CalculateField_management(layerName, "SERVICE_LOCATION.MEMBER_NUMBER", '!MEMBERNO!', "PYTHON")

log(logging.info, "Complete")
0 Kudos
MichaelVolz
Esteemed Contributor
Chris:

When is the feature class being saved to disk, as I do not see this in your code sample?

arcpy.CalculateField_management(layerName, "SERVICE_LOCATION.MEMBER_NUMBER", '!MEMBERNO!', "PYTHON")

This appears to be incorrect as layerName should be the in_table
expression should be !CIS_Account_Data.MEMBERNO! as you need to explicitly specify that this is coming from a join field.
0 Kudos
ModernElectric
Frequent Contributor
Chris:

When is the feature class being saved to disk, as I do not see this in your code sample?

arcpy.CalculateField_management(layerName, "SERVICE_LOCATION.MEMBER_NUMBER", '!MEMBERNO!', "PYTHON")

This appears to be incorrect as layerName should be the in_table
expression should be !CIS_Account_Data.MEMBERNO! as you need to explicitly specify that this is coming from a join field.


No Dice.... I am working with just this (1) field and after waiting 20 minutes - it does not complete. Is is possible this temp feature layer is causing issues??
0 Kudos
MichaelVolz
Esteemed Contributor
Chris:

Can you try to create a new layer on disk and work with that instead of the FeatureLayer?  I'm not sure this is possible, as you might have to work with the in-memory feature layer and then save to disk once all the processing has been completed.
0 Kudos