topic Re: Push list of field names from validator into script tool in Python Questions

Push list of field names from validator into script tool

Nicole_Ueberschär — Fri, 10 Dec 2021 22:12:35 GMT

Hi folks,

with help of https://community.esri.com/people/xander_bakker/blog/2016/07/19/implementing-cascading-drop-down-lists-in-a-toolbox-using-validation-with-python I gained some understanding how I can make my script tool take conditions and additional information to fill the tool.

I have three input parameters (a directory, a min and a max value). I would like to have a multiple choice list of those indicators in my txt file (in that directory) that are not empty.

My validator should check the first txt file in that directory, read the field names, if they are not within a list of excluded field names then read the first value, if it is not None than add the field name to a list. Works perfectly in pyscripter.

And I also get the correct list displayed in my tool script. Buuut, the tool says the parameter's field type is invalid after displaying the field names. I could imagine that it has something to do with the filter in the parameter which is set to DataType=Field but of course it is not really a field name any longer after returning it from my script. But what else could I use? Or can I somehow tell the self.params that it is still a (list of) field name(s)?

This is my updateParameters.

def updateParameters(self):
    """Modify the values and properties of parameters before internal
    validation is performed.  This method is called whenever a parameter
    has been changed."""
    if filedir:
        arcpy.env.workspace = filedir
        list_of_files = arcpy.ListFiles("*.txt")
        firsttable=list_of_files[0]
        print(firsttable)
        print(filedir)
        complete_filename = str(filedir)+"\\"+firsttable

        print(complete_filename)
        table=arcpy.MakeTableView_management(complete_filename, "kivu_tview")


        desc = arcpy.Describe(table)
        #fieldnamesoriginal=[field.name.encode("utf-8") for field in desc.fields] ##collects all fieldnames
        fieldselectlist=[]
        for ofields in desc.fields:
            ftype=ofields.type
            fname=ofields.name.encode("utf-8")
            if ftype=="Double" or ftype=="Long":
                print(fname)
                if fname!="Profile_No" and fname!="Longitude (degrees_east)" and fname!="Latitude (degrees_north)" and fname!="Bot. Depth (m)" and fname!="Depth (m)":
                    with arcpy.da.SearchCursor(table, fname) as calcursor:
                        for calcrow in calcursor:
                            if calcrow[0] is not None:
                               fieldselectlist.append(fname)
                               break
        print(fieldselectlist)
        self.params[3].filter.list =fieldselectlist
        self.params[3].value=fieldselectlist
    return
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Any hint would be highly appreciated.

Re: Push list of field names from validator into script tool

RandyBurton — Thu, 21 Feb 2019 07:20:42 GMT

I want to make sure I understand your work flow. You are selecting the first text file in a directory. Is this the only text file in the directory, or do you plan to process other files in the same directory? I'm curious why you are not setting up the tool to use a text file as the first parameter in place of a folder location.

For the "calcrow[0] is not None" test, is this for just the first row, or for any row in that field?

Can it be assumed that the min and max value will be selected from one of the fields that is in the field selected list?

Re: Push list of field names from validator into script tool

Nicole_Ueberschär — Thu, 21 Feb 2019 10:08:46 GMT

Thanks for looking into it Randy.

I only select the first file in the directory because the structure should be for all files inside the directory the same. They are all then processed within a seperate step (in the main python script). I could also use the table as a parameter and take its path to find the directory to process all the files, that is correct. [Is there a way to avoid the table view to be added to the map document and to be stored as a seperate process in the results window?]

I only need to check for the first row because if the value of the first row is none the whole column will be empty.

The min and max values are specified by the user, they are not derived from the parameters but mean the depth range for the parameters to be extracted.

From the error message I get, I assume I would have either to store my list of values in a different "format" than a list OR I have to set the input Data Type (which is field at the moment) differently but I can't figure out either way.

Re: Push list of field names from validator into script tool

RandyBurton — Fri, 10 Dec 2021 22:12:38 GMT

I put your code in a debugging template found on this page: Debugging a ToolValidator class. I only made a few changes, mostly using self.params[0], etc. to access the parameters. I kept the MakeTableView because it aided in identifying the field types. It might be possible to read the text file's first line directly to get the field names, and the second line to determine field types. Print statements will work in the debug script, but they will not display once it is placed in the tool's validator section (and could cause errors). For this reason, they have been commented out.

My first tool parameter was a folder type, the second was a string (for the field names). Here's my debug test code:

import arcpy

# Load the toolbox and get the tool's parameters, using the tool
#  name (not the tool label).
#
arcpy.ImportToolbox(r"C:\Path\to\tool\box\dropdown.tbx")
params = arcpy.GetParameterInfo("DropdownTest")
# refresh toolbox in Catalog or ArcMap if changes made to tool/parameters
# also close and restart Python IDE

# Set required parameters

# input, folder
params[0].value = r"C:\Users\Randy\Documents\ArcGIS\PythonScripts\test\text2field\encoding"
# input, string ['Temperature (\xc2\xb0C)', 'pH', 'Chl_A (\xc2\xb5g/L)']
# params[1].value = 'Chl_A (\xb5g/L)' # for 1 value
params[1].values = [ 'pH', 'Chl_A (\xb5g/L)' ] # for multiple values

# ToolValidator class block ------------------------------------------------------------
#
class ToolValidator(object):
    """Class for validating a tool's parameter values and controlling
    the behavior of the tool's dialog."""

    def __init__(self):
        """Setup arcpy and the list of tool parameters."""
        self.params = arcpy.GetParameterInfo()

    def initializeParameters(self):
        """Refine the properties of a tool's parameters.  This method is
        called when the tool is opened."""
        return

    def updateParameters(self):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""

        if self.params[0].altered:
            if self.params[0].value: # a folder has been entered in first parameter
                arcpy.env.workspace = filedir = self.params[0].value

                list_of_files = arcpy.ListFiles("*.txt")
                firsttable=list_of_files[0]
                # print(firsttable)
                # print(filedir)
                complete_filename = str(filedir)+"\\"+firsttable
                # print(complete_filename)

                if arcpy.Exists("kivu_tview"): #  if tableview exists then delete it
                    arcpy.Delete_management("kivu_tview")
                table=arcpy.MakeTableView_management(complete_filename, "kivu_tview")
            
                desc = arcpy.Describe(table)
                # fieldnamesoriginal=[field.name for field in desc.fields] # collects all fieldnames
                # print "Table view field names: "+str(fieldnamesoriginal)
                fieldselectlist=[]
                for ofields in desc.fields:
                    ftype=ofields.type
                    fname=ofields.name
                    if ftype in ["Double","Integer","Long"]:
                        # print fname
                        if fname not in ["Profile_No","Longitude (degrees_east)","Latitude (degrees_north)","Bot. Depth (m)","Depth (m)"]:
                            with arcpy.da.SearchCursor(table, fname) as calcursor:
                                for calcrow in calcursor:
                                    if calcrow[0] is not None:
                                       fieldselectlist.append(fname)
                                       break
                # print fieldselectlist
                arcpy.Delete_management("kivu_tview","Table View") # delete table view

                # for multiple choice
                try:
                    if self.params[1].values: #if this parameter has seleted values
                        oldValues = self.params[1].values #set old values to the selected values
                except Exception:
                    pass
                self.params[1].filter.list = sorted(fieldselectlist) #set the filter list equal to the sorted values
                newValues = self.params[1].filter.list
                try:
                    if len(oldValues): # if some values are selected
                        self.params[1].values = [v for v in oldValues if v in newValues] # check if seleted values in new list,
                        # if yes, retain the seletion.
                except Exception:
                    pass

                # for single choice
                # self.params[1].filter.list = sorted(fieldselectlist) # set second parameter filter list
                # if self.params[1].value not in self.params[1].filter.list:
                #     self.params[1].value = self.params[1].filter.list[0] # default to first item in list

        return

    def updateMessages(self):
        """Modify the messages created by internal validation for each tool
        parameter.  This method is called after internal validation."""
        return

# Call routine(s) to debug ------------------------------------------------------------------------------------------
#
validator = ToolValidator()
validator.updateParameters()
validator.updateMessages()

# testing results
print(arcpy.GetMessages())
print u"Folder: {}".format(params[0].value)
# print u"Fields: {}".format(params[1].value) # for 1 value
print u"Fields: {}".format(params[1].values) # for multiple values
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Once I was happy with the debug test, I pasted the ToolValidator section into the tool's validator.

Hope this helps.

Re: Push list of field names from validator into script tool

Nicole_Ueberschär — Fri, 10 Dec 2021 22:12:40 GMT

Thanks Randy but it still not returning the field names in the required way. This is only writing "Short" as an option to my choice list and also this has the wrong data type because it is not a field name. I don't even know where "Short" is coming from.

I am debugging in PyScripter and everything works fine except of pushing the values back. I need a multiple values list to select my field names from.

Ha, I changed the data type to string and then the tool accepts the input but when executing the tool!

I passed the list as it is:

        self.params[3].filter.list=sorted(fieldselectlist)
        self.params[3].value=sorted(fieldselectlist)
‍‍

I'm not sure though if the first line is even necessary. But it works. Now I just have to adjust the main script to "eat" my list...

Re: Push list of field names from validator into script tool

Nicole_Ueberschär — Fri, 22 Feb 2019 09:01:02 GMT

Only that I get now an error message when trying to execute the tool:

If I interprete the line numbers correctly (does it count in the comments?), line 4 should be list_of_files = arcpy.ListFiles("*.txt") (or around there) and line 51 would be the last line. But I can click OK and then my main script starts running.

It still behaves a bit funny because it would not allow me to "unselect all fields". Also when I unselected a couple of field names then they would be turned back on when clicking "OK" to start the tool.

The same the other way around: When I put the value='' then it would not allow me to make a selection.

Not perfect yet...

Re: Push list of field names from validator into script tool

Nicole_Ueberschär — Fri, 22 Feb 2019 09:30:54 GMT

I figured out that I should leave the value unset (so just remove the last line before return) to not preset a value.

But I still get a "Bad file descriptor" error message...

Re: Push list of field names from validator into script tool

Nicole_Ueberschär — Fri, 22 Feb 2019 11:33:49 GMT

Funnily, sometimes I get the "Bad file descriptor" error message, sometimes I don't. But most of the times the tool is running properly.

Re: Push list of field names from validator into script tool

RandyBurton — Mon, 25 Feb 2019 02:35:49 GMT

I posted updated validator code above. I hadn't considered that you are using a multivalue selection list. In addition, there is a check that needed to be done to see if any fields were selected. I also updated the single selection check but commented it out.

Since some of your fields are using extended characters, those field name parameters will be inside single quotes.

Re: Push list of field names from validator into script tool

Nicole_Ueberschär — Mon, 25 Feb 2019 06:31:13 GMT

Thanks a lot!

Re: Push list of field names from validator into script tool

Nicole_Ueberschär — Mon, 25 Feb 2019 13:01:56 GMT

Is there a way to enable the "Select all/unselect all" functionality for the tool?

I was able to "select all" after loading the list through the validator but the buttons inside the tool don't work. I guess that would another "on change" event but how do I catch that?

Re: Push list of field names from validator into script tool

RandyBurton — Mon, 25 Feb 2019 17:54:15 GMT

I had used the select all button the tool interface provides without a problem. I did not try the deselect button but will check it out later.

Re: Push list of field names from validator into script tool

Nicole_Ueberschär — Mon, 25 Feb 2019 18:01:06 GMT

Strange... For me both don't do anything.

Re: Push list of field names from validator into script tool

RandyBurton — Mon, 25 Feb 2019 19:19:19 GMT

And to confirm, you replaced the validator code with the appropriate section from above and then refreshed the tool?

Re: Push list of field names from validator into script tool

Nicole_Ueberschär — Fri, 10 Dec 2021 22:12:43 GMT

As far as I can see, yes. I added this to have all selected: self.params[3].value=self.params[1].filter.list after your line 79.

I also had to add another loop to go through all tables in the folder. It turned out (in contrast to what I thought) that not all tables have data for the same indicators... This is my updated validation code. Note, that I have params[3] in the end... Ah, yes, and I added a lower case sort of the field names.

  def updateParameters(self):
    """Modify the values and properties of parameters before internal
    validation is performed.  This method is called whenever a parameter
    has been changed."""
    filedir=self.params[0].value
    if filedir:
        arcpy.env.workspace = filedir
        list_of_files = arcpy.ListFiles("*.txt")
        firsttable=list_of_files[0]

        for txtfile in list_of_files:
            complete_filename = str(filedir)+"\\"+str(txtfile)
            table=arcpy.MakeTableView_management(complete_filename, "kivu_tview")
            desc = arcpy.Describe(table)
            arcpy.AddMessage("Table to be analyzed: "+str(table))
            fieldselectlist=[]         #Will collect the field names
            for ofields in desc.fields:
                ftype=ofields.type
                fname=ofields.name.encode("utf-8")
                if ftype in ["Double","Long"]:
                    #print(fname)
                    if fname not in ["Profile_No","Longitude (degrees_east)" ,"Latitude (degrees_north)" ,"Bot. Depth (m)","Depth (m)","Depth (0.5 m )" ,"Depth (0.5 m)","Distance from platform","Time (actual)"]:
                        with arcpy.da.SearchCursor(table, fname) as calcursor:
                            for calcrow in calcursor:
                                if calcrow[0] is not None:
                                    if calcrow[0] not in fieldselectlist:
                                        fieldselectlist.append(fname)

                                    break
                                arcpy.AddMessage(fieldselectlist)
            arcpy.Delete_management("kivu_tview","Table View") # delete table view
        #sorted_list=fieldselectlist.sort(key=str.lower)
        #sorted_list=fieldselectlist.sort(key=lambda x:x.lower())
        if 'pH' in fieldselectlist and 'pH corrected' in fieldselectlist:
              fieldselectlist.remove('pH')
        if 'Dissolved Oxygen (mg/L)' in fieldselectlist and 'Dissolved Oxygen corrected (mg/L)' in fieldselectlist:
              fieldselectlist.remove('Dissolved Oxygen (mg/L)')


        sorted_list_not_cs=sorted(fieldselectlist,key=str.lower)

        arcpy.AddMessage("sortiert, not case sensitive: "+str(sorted_list_not_cs))



        # for multiple choice
        try:
            if self.params[3].values: #if this parameter has seleted values
                oldValues = self.params[3].values #set old values to the selected values
        except Exception:
            pass
        self.params[3].filter.list = sorted_list_not_cs #set the filter list equal to the sorted values
        newValues = self.params[3].filter.list
        self.params[3].value=self.params[3].filter.list
        try:
            if len(oldValues): # if some values are selected
                self.params[3].values = [v for v in oldValues if v in newValues] # check if seleted values in new list,
                # if yes, retain the seletion.
        except Exception:
            pass

        #self.params[3].filter.list=sorted(fieldselectlist)

    return‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍