topic Re: Wait for AddField to finish until running CalculatField in Python Questions

Wait for AddField to finish until running CalculatField

TimWitt2 — Sat, 11 Dec 2021 15:33:29 GMT

Hey everybody,

I am creating python toolbox and one of the tools adds a bunch of fields to a feature class that the user chooses and after wards calculates those fields with values from the original feature class. I run into the issue that the field calculations can't run because the fields haven't been added yet.

This is the code I use:

import arcpy  
  
class Toolbox(object):  
    def __init__(self):  
        """Define the toolbox (the name of the toolbox is the name of the .pyt file)."""  
        self.label = "My first python toolbox"  
        self.alias = "Examples"  
  
        # List of tool classes associated with this toolbox  
        self.tools = [Tool1]  
  
class Tool1(object):  
    def __init__(self):  
        """Define the tool (tool name is the name of the class)."""  
        self.label = "Nena Street Tool"  
        self.description = "Example showing how to create a value table tool with drop downs"  
        self.canRunInBackground = False  
  
    def getParameterInfo(self):  
        """Define parameter definitions"""  
  
        #  
        # Define Parameter 0  
        #  
  
        # Param0 will be a FeatureLayer 
        param0 = arcpy.Parameter(displayName = "Your Street Layer",name="Site_layer",datatype="GPFeatureLayer",parameterType="Required",direction="Input")          

        #  
        # Define Parameter 1  
        #  
  
        param1 = arcpy.Parameter(displayName = "",name="Activity_layers",datatype="GPValueTable",parameterType="Optional", direction="Input")  
  
 
        param1.columns=[["String","NENA Field"],["String","Your Field"]]  
  
  
        # This says the filter on the second column will be a value List  
        param1.filters[1].type="ValueList"
        param1.value = "Source; Updatedate; Effective; Expire; RCL_NGUID; AdNumPre_L; AdNumPre_R; FromAddr_L; ToAddr_L; FromAddr_R; ToAddr_R; Parity_L; Parity_R"          
  
        # This adds a dummy value to the list, this will get overwritten by the updateParameters function  
        param1.filters[1].list=["Please Select a street layer"]  
  
        # Create a list of parameters and return  
        params = [param0,param1]  
        return params  
  
    def isLicensed(self):  
        """Set whether tool is licensed to execute."""  
        return True  
  
    def updateParameters(self, parameters):  

  
        if parameters[0].altered:
          fields = arcpy.ListFields(parameters[0].value)
          mylist = []
          for field in fields:
             mylist.append(field.name)
          parameters[1].filters[1].list = mylist
        return  
  
    def updateMessages(self, parameters):  
        """Modify the messages created by internal validation for each tool 
        parameter.  This method is called after internal validation."""  
        return  
  
    def execute(self, parameters, messages):  
        """The source code of the tool."""  
        try:  
            # Some code to show that I had managed to get what I wanted intot the ValueTable               
            data = parameters[1].values
            arcpy.AddField_management(parameters[0].value, "Source", "TEXT", "", "", "75", "", "NULLABLE", "NON_REQUIRED", "")
            arcpy.AddField_management(parameters[0].value, "DateUpdate", "DATE", "", "", "", "", "NULLABLE", "NON_REQUIRED", "")
            arcpy.AddField_management(parameters[0].value, "Effective", "DATE", "", "", "", "", "NULLABLE", "NON_REQUIRED", "")
            search = 'Source'
            for sublist in data:
              if sublist[0] == search:
                final = sublist
                mysource = final[1]
                sourceData = "[" + mysource + "]"
                arcpy.CalculateField_management(parameters[0].value, "Source", sourceData, "VB", "")
                break

        except arcpy.ExecuteError:  
            messages.addErrorMessage(arcpy.GetMessages()) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

In line 75 I add the "Source" field and in line 84 I try to calculate it, but I get the error that the "Source" field does not exist.

Any help would be appreciated.

Tim

Re: Wait for AddField to finish until running CalculatField

JoshuaBixby — Fri, 14 Apr 2017 13:20:13 GMT

And you have verified it gets created, eventually? If you comment out Line 84 and run the script, does the table look the way you expect?

If everything is working as expected and this is simply a matter or timing, a simplistic way to deal with it would be to put a time.sleep() line after the final field addition. You would have to balance sleeping long enough to have the field created and waiting too long so the user gets frustrated.

I would start with the simple approach of inserting time.sleep(10). If the code still errors out than I think that points to something not working quite right with creating the fields.

Re: Wait for AddField to finish until running CalculatField

TimWitt2 — Fri, 14 Apr 2017 13:26:23 GMT

Joshua,

thanks for the quick reply. Yes the field is created and when I run the tool again it gets populated. I will have to add a total of 40 fields. That might be enough time for the first field to be available? But I will try the time.sleep approach first.

Thanks,

Tim

Re: Wait for AddField to finish until running CalculatField

TimWitt2 — Fri, 14 Apr 2017 13:30:40 GMT

Sadly it didn't work. I have a feeling that I need to somehow update the parameters[0].value before I run the CalculateField.

Re: Wait for AddField to finish until running CalculatField

DanPatterson_Retired — Fri, 14 Apr 2017 15:03:58 GMT

vaguely remember something about using featurelayer vs featureclass and schema being modified which takes time blah blah. It made more sense that this, but I seem to recollect that pairing the creation then the calculation for one field before proceeding on to the next pairing provided the necessary break. Worth a shot

Re: Wait for AddField to finish until running CalculatField

TimWitt2 — Fri, 14 Apr 2017 15:36:36 GMT

In line 84 I replaced parameters[0].value with parameters[0].valueAsText and now it is working. I assume parameters[0].value is the original feature layer without the extra fields and parameters[0].valueAsText grabs the feature layer with the new field.

Re: Wait for AddField to finish until running CalculatField

curtvprice — Sat, 11 Dec 2021 15:33:32 GMT

Thanks for sharing your fix Tim. I think in the case of your layer you are passing, .valueAsText gives you the string label of the layer, which is more likely to play well with the tools and their interactions with the layer in memory.

.valueAsText [GetParameterAsText() for script tools] is (IMHO) the safe way to pass parameters to "go with the flow" how GP is implemented. I only use .value [GetParameter()] to access certain kinds of complex object parameters that are easier to work with directly as objects (value tables come to mind).

Layers are such an odd duck - I wish Esri could have designed this to give us an session-unique integer handle instead of depending on strings that may or not match the layer we want to access. I'm sure there are many historical reasons things are they way the are in arcpy. So you have to be careful with layers.

For code like yours, I make a local layer from the input. If in_fc is a feature class, not a layer, the multiple tools you run on it will run faster because the layer schema (unlike a dataset) are pre-validated, the tool has less to do before it runs, as the data a layer points to is assumed to exist without an additional check. Also the tool parameter validation can ask the (fast) layer in memory whether the field is there without going back to disk (slow) to verify the field there.

in_fc = parameters[0].valueAsText
lyrFC = arcpy.MakeFeatureLayer_management(in_fc, "_infc")
arcpy.AddField_management(lyrFC, "FOO", "LONG")
arcpy.CalculateField_management(lyrFC, "FOO", -1)
arcpy.Delete_management(lyrFC) # clean up‍‍‍‍‍

Re: Wait for AddField to finish until running CalculatField

TimWitt2 — Mon, 17 Apr 2017 12:58:26 GMT

Thanks for the hint Curtis, since I will have to do that for 40 fields I am looking for any way to speed up my code.

Re: Wait for AddField to finish until running CalculatField

BruceHarold — Mon, 17 Apr 2017 14:07:10 GMT

arcpy.da.ExtendTable is a way to quickly add fields, below some lines from a tool of mine:

arcpy.CreateFeatureclass_management(outPath,outName,'POINT',
                                    spatial_reference=sR)
narray = numpy.array([], numpy.dtype([('_ID', numpy.int),
                                      ('ResultID',numpy.long),
                                      ('Loc_name','|S23'),
                                      ('Status','|S1'),
                                      ('Score',numpy.double),
                                      ('Match_addr','|S300'),
                                      ('Addr_type','|S50'),
                                      ('AddNum','|S80'),
                                      ('AddNumFrom','|S10'),
                                      ('AddNumTo','|S10'),
                                      ('Side','|S1'),
                                      ('StPreDir','|S20'),
                                      ('StPreType','|S50'),
                                      ('StName','|S100'),
                                      ('StType','|S30'),
                                      ('StDir','|S20'),
                                      ('StAddr','|S300'),
                                      ('Nbrhd','|S80'),
                                      ('City','|S120'),
                                      ('Subregion','|S120'),
                                      ('Region','|S120'),
                                      ('Postal','|S20'),
                                      ('PostalExt','|S70'),
                                      ('Country','|S30'),
                                      ('LangCode','|S10'),
                                      ('Distance',numpy.double),
                                      ('X',numpy.double),
                                      ('Y',numpy.double),
                                      ('DisplayX',numpy.double),
                                      ('DisplayY',numpy.double),
                                      ('Xmin',numpy.double),
                                      ('Xmax',numpy.double),
                                      ('Ymin',numpy.double),
                                      ('Ymax',numpy.double)]))
arcpy.da.ExtendTable(outFC,arcpy.Describe(outFC).OIDFieldName,narray,"_ID")
arcpy.AlterField_management(outFC,'Nbrhd',new_field_alias='Neighborhood')

Re: Wait for AddField to finish until running CalculatField

TimWitt2 — Mon, 17 Apr 2017 14:54:50 GMT

Bruce thanks for the code! How would I create a date field?

Re: Wait for AddField to finish until running CalculatField

curtvprice — Mon, 17 Apr 2017 15:06:07 GMT

You're welcome!

I will have to do that for 40 fields I am looking for any way to speed up my code.

Given your application (it sounds like you need to populate these fields from existing fields with different names, maybe parts of field content too), you may want to consider setting up a field map, add your 40 fields to it, and running Feature Class To Feature Class. Field maps are a bit tricky to work with but I'm certain this is the fastest way to add 40 fields to a table and copy content over from existing fields.

That ExtendTable thing is awesome though!