Select to view content in your preferred language

Python code block in calculate field problem

3258
5
08-24-2010 01:09 PM
ChrisMathers
Deactivated User
Ok I am having a problem with a python code block. It works for about 1300 rows and then fails without error. The calculate field tool runs for 15-20 minutes and then without returning any error returns to the shell. The problem is stopping the whole script without giving any sort of error. Also it seems to be skipping value.replace(\"'\",'') though I think that the need to escape those two " are confusing the GP. Perhaps there is a better way to convert a string to a set
 
expression='getvalue(!parce_test_spatial_join.FLD_ZONE!)'
codeblock="def getvalue(string):\n\
    valuelist=string.split(',')\n\
    valueset=set(valuelist)\n\
    value=str(valueset)[5:-2]\n\
    value.replace(\"'\",'')\n\
    return value"


I am thinking of replacing the whole code block with this but I still have to escape those quotes or python gets mad.

codeblock="def(string):\n\
    value=str(set(string.split(','))).replace(\"'\",'')[5:-2]\n\
    return value"
0 Kudos
5 Replies
KimOllivier
Honored Contributor
You don't need all the complications of defining a function as a string. Just create the function in the codeblock and use a simple expression that uses the function. Indents and extra lines will be honoured.
Calculator is very bad at handling data errors. If your expressions result in an exception, then you won't get any feedback. Better to use a cursor in a script and trap the errors so you can see the faulty data. It is just as fast, because a cursor implements your expression anyway.


codeblock that goes in the box :
def getvalue(string):
    valuelist=string.split(',')
    valueset=set(valuelist)
    value=str(valueset)[5:-2]
    value.replace('"','') # not sure what you are trying to replace - a double quote?
    return value


expression in the box:
getvalue(!parce_test_spatial_join.FLD_ZONE!)


Maybe a python script equivalent:
import arcpy
cur = arcpy.UpdateCursor("parce_test_spatial_join")
try:
    for row in cur:
        value = row.fld_zone
        valuelist=string.split(',')
        valueset=set(valuelist)
        value=str(valueset)[5:-2]
        value.replace('"','')
        row.target_field = value # enter your own field here
        cur.updateRow(row)
    del cur
except :
    print "failure on ",value
0 Kudos
ChrisMathers
Deactivated User
The reason I did the string formatting is that this code block is the parameter for the calculate field tool being called in python. I cant use a cursor here because I am joining tables and cant seem to create update cursors on joined tables. People have told me that I can do this in 9.3.1 but I always get an error when I try. the .replace() is striping out the single quotes that are added when the string was made into a list.

import arcgisscripting
gp=arcgisscripting.create(9.3)
gp.MakeFeatureLayer_management("C:/GIS Projects/CRW/TrakIt_Source_RS.gdb/Target_Data/backup_table","parce_test","#","#","A1RENUM A1RENUM VISIBLE NONE;RE_LABEL RE_LABEL VISIBLE NONE;A2OWNAME A2OWNAME VISIBLE NONE;A3MAILADDR A3MAILADDR VISIBLE NONE;A4MAILADDR A4MAILADDR VISIBLE NONE;A5MAILADDR A5MAILADDR VISIBLE NONE;A6MAILCITY A6MAILCITY VISIBLE NONE;A7MAILST A7MAILST VISIBLE NONE;A8MAILZIP A8MAILZIP VISIBLE NONE;A9MAILCTRY A9MAILCTRY VISIBLE NONE;DORAPPCODE DORAPPCODE VISIBLE NONE;DORAPPDESC DORAPPDESC VISIBLE NONE;DTAXDIST DTAXDIST VISIBLE NONE;DEXEMPTS DEXEMPTS VISIBLE NONE;DSITEADDR DSITEADDR VISIBLE NONE;AHSENO AHSENO VISIBLE NONE;AHSENOSUF AHSENOSUF VISIBLE NONE;ASTNAME ASTNAME VISIBLE NONE;ASTMD ASTMD VISIBLE NONE;ASTDIR ASTDIR VISIBLE NONE;ASTUNIT ASTUNIT VISIBLE NONE;DSITEZIP DSITEZIP VISIBLE NONE;DMAPAREA DMAPAREA VISIBLE NONE;DMAPNUM DMAPNUM VISIBLE NONE;DSECTION DSECTION VISIBLE NONE;DTAXACRES DTAXACRES VISIBLE NONE;GISACRES GISACRES VISIBLE NONE;LEGALINE1 LEGALINE1 VISIBLE NONE;LEGALINE2 LEGALINE2 VISIBLE NONE;LEGALINE3 LEGALINE3 VISIBLE NONE;LEGALINE4 LEGALINE4 VISIBLE NONE;LEGALINE5 LEGALINE5 VISIBLE NONE;LEGALINE6 LEGALINE6 VISIBLE NONE;LEGALINE7 LEGALINE7 VISIBLE NONE;LEGALINE8 LEGALINE8 VISIBLE NONE;LEGALINE9 LEGALINE9 VISIBLE NONE;LL1APPTYPE LL1APPTYPE VISIBLE NONE;LL1DEPTH LL1DEPTH VISIBLE NONE;LL1FRONT LL1FRONT VISIBLE NONE;LL1UNITS LL1UNITS VISIBLE NONE;LL1UNITYPE LL1UNITYPE VISIBLE NONE;S1APPTYPE S1APPTYPE VISIBLE NONE;S1AREAHEAT S1AREAHEAT VISIBLE NONE;S1AREATTL S1AREATTL VISIBLE NONE;S1BATHRMS S1BATHRMS VISIBLE NONE;S1BEDRMS S1BEDRMS VISIBLE NONE;S1STORIES S1STORIES VISIBLE NONE;S1UNITS S1UNITS VISIBLE NONE;S1YRBLTACT S1YRBLTACT VISIBLE NONE;S1YRBLTEFF S1YRBLTEFF VISIBLE NONE;SALE1DATE SALE1DATE VISIBLE NONE;SALE1GRNTR SALE1GRNTR VISIBLE NONE;SALE1GRTEE SALE1GRTEE VISIBLE NONE;SALE1INST SALE1INST VISIBLE NONE;SALE1PRADJ SALE1PRADJ VISIBLE NONE;SALE1QU_VI SALE1QU_VI VISIBLE NONE;SALE2DATE SALE2DATE VISIBLE NONE;SALE2GRNTR SALE2GRNTR VISIBLE NONE;SALE2GRTEE SALE2GRTEE VISIBLE NONE;SALE2INST SALE2INST VISIBLE NONE;SALE2PRADJ SALE2PRADJ VISIBLE NONE;SALE2QU_VI SALE2QU_VI VISIBLE NONE;SALE3DATE SALE3DATE VISIBLE NONE;SALE3GRNTR SALE3GRNTR VISIBLE NONE;SALE3GRTEE SALE3GRTEE VISIBLE NONE;SALE3INST SALE3INST VISIBLE NONE;SALE3PRADJ SALE3PRADJ VISIBLE NONE;SALE3QU_VI SALE3QU_VI VISIBLE NONE;VAPBLDG VAPBLDG VISIBLE NONE;VAPFEATXTR VAPFEATXTR VISIBLE NONE;VAPLNDAGR VAPLNDAGR VISIBLE NONE;VAPLNDMRK VAPLNDMRK VISIBLE NONE;VAPTOTAL VAPTOTAL VISIBLE NONE;VASCAPPED VASCAPPED VISIBLE NONE;VASEXEMPT VASEXEMPT VISIBLE NONE;VASJUST VASJUST VISIBLE NONE;VASTAXABLE VASTAXABLE VISIBLE NONE;VASTOTAL VASTOTAL VISIBLE NONE;Flood_zn Flood_zn VISIBLE NONE;Co_FLU Co_FLU VISIBLE NONE;Co_Zoning Co_Zoning VISIBLE NONE;Eco_sys Eco_sys VISIBLE NONE;H_evac H_evac VISIBLE NONE;Elem_zone Elem_zone VISIBLE NONE;Midd_zone Midd_zone VISIBLE NONE;High_zone High_zone VISIBLE NONE;PO_name PO_name VISIBLE NONE;Serv_area Serv_area VISIBLE NONE;Sub_name Sub_name VISIBLE NONE;Sub_blk Sub_blk VISIBLE NONE;Sub_lot Sub_lot VISIBLE NONE;Road_dist Road_dist VISIBLE NONE;Surge Surge VISIBLE NONE;Comm_Dist Comm_Dist VISIBLE NONE;Vote_Prcnct Vote_Prcnct VISIBLE NONE;Fire_Dist Fire_Dist VISIBLE NONE;Shape_Length Shape_Length VISIBLE NONE;Shape_Area Shape_Area VISIBLE NONE")
gp.addjoin_management('parce_test', "A1RENUM", r"C:\GIS Projects\CRW\TrakIt_Source_RS.gdb\parce_test_spatial_join", "A1RENUM", "KEEP_ALL")
cursor=gp.updatecursor('parce_test')
row=cursor.Next()
while row:
    print row.getvalue('[parce_test_spatial_join.FLD_ZONE]')
 
>>> 
Traceback (most recent call last):
  File "C:/GIS Projects/CRW/test.py", line 5, in <module>
    cursor=gp.updatecursor('parce_test')
RuntimeError: ERROR 999999: Error executing function.
0 Kudos
RichardFairhurst
MVP Honored Contributor
Python absolutely sucks with joined data.  Python is up to 5000% slower with a join in place than VB script, so you should absolutely avoid doing calculations using Python with a join in place whenever possible.  Joins and the field calculator should only be used in scripts where a data transfer is occurring from the child table to the parent table (and if it is a simple transfer use VB Script syntax).  I don't see that your calculation in this case is dependent on the join (but I may be missing something).  Joins with Python also can cause out ot memory errors that cause unexpected failures as you can see.

If this is scripted and the calculation you want to do is possible outside of the join reorganize your operations to do the calculation outside the join and then create the join.  Everything will perform much much faster and failure will be minimized.  If the join is necessary for transferring data, do just that operation and then break the join to do any other operations that transform the data.  You will save an immense amount of time.  (Python is necessary for advanced expressions, but never perform them with a join in place).  Breaking the transfer and the transformation into separate operations will actually save you time if you manage teh joins so that they are removed prior to and advanced calculations.

ESRI has confirmed the slow performance of Python on joined tables and has no fix at either ArcGIS 9.3.1 or ArcGIS 10.  VB Script syntax always performs faster for simple transfers between joined tables, so for any simple transfers use VB syntax (simple means only without any VB specific functions or methods, just [field] tranfer syntax).  The VB syntax on joined tables always takes advantage of indexed join fields, but the Python syntax does not seem to, which is why the calculations are always faster with VB script.  (Of course this means you should create indexes on your joined fields before doing the calculations and use VB script syntax)

I also have never been able to use an updatecursor on a joined table, so I don't know of a way to make that work (seems like ESRI somehow does it, but they have never indicated how as far as I have seen).

Hope this helps.
0 Kudos
KimOllivier
Honored Contributor
You could look at the MakeQueryTable tool. This will effectively join a number of tables very efficiently. I have tried it with extremely large tables that would fail with a Join.
Haven't tried a cursor on the resulting layer....
0 Kudos
ChrisMathers
Deactivated User
Thanks to both of you. After much deliberation last night I decided "ef it Ill just forget the set()." The whole point of this was that the output of a spatial join with the important field set to Join as its merge rule gave for some features things like 'x,x,x,a,ae,ve,ve,ve,a,ae,x,x.' I wanted to make that a set so that only the unique values copied over to the other table. At this point though Ill just copy the whole string and be done with it. It was more of an aesthetic change anyways. Not being able to use an update cursor on join tables is very irritating though.
0 Kudos