Running a .pyt throws ERROR 001683: Found Python 2 to 3 errors for syntax that does not cause an error on syntax check, or other python environments

[gis_bCapell]

I am developing .pyt Python Tools with pre-existing code that I am porting into the .pyt format.

The Python code works when scripts are executed within Custom Toolboxes (.atbx), and as standalone python scripts from command prompt or my IDE (PyCharm). When I include some custom classes and functions in the .pyt from the scripts and run I get this error: ERROR 001683: Found Python 2 to 3 errors. Below is the error message and concise reproducible .pyt examples of the type of issue demonstrating error and success (following recommended edit).

I also tested the relevant error and success snippets in an .ipynb where both return the same output. Why does the .pyt error out from this code difference when the .ipynb and other Python environments do not?

success .pyt:

# -*- coding: utf-8 -*-

import arcpy


class Toolbox:
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = "toolbox"

        # List of tool classes associated with this toolbox
        self.tools = [Tool]

class Tool:
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "success"
        self.description = ""

    def getParameterInfo(self):
        """Define the tool parameters."""
        params = []
        return params

    def isLicensed(self):
        """Set whether the tool is licensed to execute."""
        return True

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return

    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter. This method is called after internal validation."""
        return

    def execute(self, parameters, messages):
        """The source code of the tool."""
        main()
        return

    def postExecute(self, parameters):
        """This method takes place after outputs are processed and
        added to the display."""
        return

d_test = {
    '1':1,
    '2':2
}


def main():
    for item in list(d_test.values()): #line not causing error
        print(item)
        arcpy.AddMessage(str(item))

error .pyt:

# -*- coding: utf-8 -*-

import arcpy


class Toolbox:
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = "toolbox"

        # List of tool classes associated with this toolbox
        self.tools = [Tool]

class Tool:
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "error"
        self.description = ""

    def getParameterInfo(self):
        """Define the tool parameters."""
        params = []
        return params

    def isLicensed(self):
        """Set whether the tool is licensed to execute."""
        return True

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return

    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter. This method is called after internal validation."""
        return

    def execute(self, parameters, messages):
        """The source code of the tool."""
        main()
        return

    def postExecute(self, parameters):
        """This method takes place after outputs are processed and
        added to the display."""
        return

d_test = {
    '1':1,
    '2':2
}


def main():
    for item in d_test.values(): #line causing error
        print(item)
        arcpy.AddMessage(str(item))

[HaydenWelch]

You need to move the d_test definition into the main loop. It's being shadowed currently.

Ideally you just treat the execute() method of the Tool Class as your main():

# -*- coding: utf-8 -*-

import arcpy


class Toolbox:
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = "toolbox"

        # List of tool classes associated with this toolbox
        self.tools = [Tool]

class Tool:
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "error"
        self.description = ""

    def getParameterInfo(self):
        """Define the tool parameters."""
        params = []
        return params

    def isLicensed(self):
        """Set whether the tool is licensed to execute."""
        return True

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return

    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter. This method is called after internal validation."""
        return

    def execute(self, parameters, messages):
        """The source code of the tool."""
        d_test = {
            '1':1,
            '2':2
        }
        for item in d_test.values(): #line causing error
            print(item)
            arcpy.AddMessage(str(item))
        return

    def postExecute(self, parameters):
        """This method takes place after outputs are processed and
        added to the display."""
        return

If you need to feed that d_test value to the tool from another script, you can run that script from the __init__ function and use that in your execute loop, but remember that the __init__ is only run when the tool is initially opened, while the execute() is run every time the Run button is clicked:

# -*- coding: utf-8 -*-

import arcpy


class Toolbox:
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = "toolbox"
        
        # List of tool classes associated with this toolbox
        self.tools = [Tool]

class Tool:
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "error"
        self.description = ""
        # Precalculated dictionary
        self.d_test = {
            '1':1,
            '2':2
        }

    def getParameterInfo(self):
        """Define the tool parameters."""
        params = []
        return params

    def isLicensed(self):
        """Set whether the tool is licensed to execute."""
        return True

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return

    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter. This method is called after internal validation."""
        return

    def execute(self, parameters, messages):
        """The source code of the tool."""
        # Pulls dictionary that is initialized on tool open
        for item in self.d_test.values():
            print(item)
            arcpy.AddMessage(str(item))
        
        d_test = {
            '3':3,
            '4':4
        }
        # Pulls dictionary that is initialized in the execute method
        for item in d_test.values():
            print(item)
            arcpy.AddMessage(str(item))
        
        return

    def postExecute(self, parameters):
        """This method takes place after outputs are processed and
        added to the display."""
        return

Ideally with pyt files, each tool will be self contained and all the variables and methods needed for that tool should be defined as tool class methods. You can also create a module and do relative script imports at the top of your toolbox so you can have some general methods that all tools can access:

#Toolbox Folder
#| -- toolbox.pyt
#| --helpermodule
#    | -- __init__.py
#    | -- helpermodule.py

Toolbox.pyt

# -*- coding: utf-8 -*-

import arcpy
from helpermodule.helpermodule import HelperModule

class Toolbox:
    def __init__(self):
        ...

class Tool:
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "error"
        self.description = ""
        # Precalculated dictionary
        self.d_test = HelperModule().get_test_dict()

    def getParameterInfo(self):
        ...

    def isLicensed(self):
        ...

    def updateParameters(self, parameters):
        ...

    def updateMessages(self, parameters):
        ...

    def execute(self, parameters, messages):
        for item in self.d_test.values():
            print(item)
            arcpy.AddMessage(str(item))
        
        d_test = HelperModule().get_test_dict()
        for item in d_test.values():
            print(item)
            arcpy.AddMessage(str(item))
        return

    def postExecute(self, parameters):
        ...

helpermodule.py

class HelperModule:
    def __init__(self):
        self.d_test = {"a": 1, "b": 2, "c": 3}

    def get_test_dict(self):
        return self.d_test

[gis_bCapell]

Hi @HaydenWelch ,

Thanks for the suggestions. I tried implementing d_test.values() when starting a for loop within the Tool.execute() definition (as you suggested), and within a new_method definition that is called (Tool.new_method()) within Tool.execute(). Both returned the same error originally posted.

Did you test your suggestions? I am interested in seeing if you get the same error. 

Can you share an Esri source describing your comment "remember that the __init__ is only run when the tool is initially opened, while the execute() is run every time the Run button is clicked"? I am seeking deeper descriptions of the Python Toolbox functionality than I have found so far. 

Thanks

[HaydenWelch]

I definitely jumped the gun on my solution, it seems that your error is coming from your Python syntax being Python 3 while the environment for your toolbox is Python 2. ArcGIS Desktop, ArcGIS Server, and ArcGIS Engine all still use Python 2 so if you're using one of those you'll need to refactor to use Python 2.

I test the toolbox code in ArcPro 3.3 and it runs fine:

# -*- coding: utf-8 -*-

import arcpy


class Toolbox:
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = "toolbox"

        # List of tool classes associated with this toolbox
        self.tools = [Tool]


class Tool:
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Tool"
        self.description = ""
        self.d_test =\
            {
                "a": 1,
                "b": 2,
                "c": 3,
            }

    def getParameterInfo(self):
        """Define the tool parameters."""
        params = None
        return params

    def isLicensed(self):
        """Set whether the tool is licensed to execute."""
        return True

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return

    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter. This method is called after internal validation."""
        return

    def execute(self, parameters, messages):
        """The source code of the tool."""
        d_test =\
            {
                "d": 4,
                "e": 5,
                "f": 6,
            }
        for i in list(d_test.values()):
            print(i)
            arcpy.AddMessage(str(i))
        
        for i in list(self.d_test.values()):
            print(i)
            arcpy.AddMessage(str(i))
        
        return

    def postExecute(self, parameters):
        """This method takes place after outputs are processed and
        added to the display."""
        return

So you just need to make sure that your arcpy environment is using Python 3

I'm not sure how your environment settings got out of whack, but that specific error is definitely telling you that there's a problem with it.

As for toolbox stuff, I've spent a few years now working with them and there really isn't much good info out there so I've been coming up with these solutions by myself. The reason I know the __init__ function is called on tool opening is through testing. I've current;y got a setup that has dynamically loaded tool modules that exploit this for logging and reporting purposes (I can run an arbitrary script when a tool is loaded and not run meaning I can have a user fill out some config before they run the tool).

I have a repo with an example modular toolbox framework if you're interested in that:

https://github.com/hwelch-fle/pytframe/tree/main

[gis_bCapell]

Hi @HaydenWelch ,

I see your successful run includes:

list(d_test.values())

 to loop through the dictionary values. That is why your code doesn't throw an error on tool execution. If you change it to:

d_test.values()

you will encounter the error. 

My python environment is default and I am running ArcGIS Pro 3.3.

To further clarify the full issue scope. I have this issue I presented with a Python 2 to 3 error caused by not passing the dictionary.values() iterator to list() so it becomes an iterable. I also have the Python 2 to 3 error in other cases while I am porting some legacy code (classes and functions) into a .pyt so it is more user friendly. Can you shed any light on the question of why the code will execute without issue in .ipynb and IDE's setup to use the default ArcGIS Pro 3.3 environment, but cause the Python 2 to 3 error when ran in .pyt?

I really appreciate you engaging me on this issue!

[HaydenWelch]

There shouldn't be any difference between list(dict.values()) and dict.values() as both return an iterable object.

I tried the test again with a raw .values(), a list cast and an iter cast and all returned the same results.

Have you tried outputting sys.version_info? Just import the sys module and add that last line to your code and see what's going on.

[gis_bCapell]

As an initial note I'll add that from my web searching there is a technical difference between list() and dict.values(): list() creates an iterable, while dict.values() returns an iterator. So that's a slight difference from your note "There shouldn't be any difference between list(dict.values()) and dict.values() as both return an iterable object."

• I am not able to successfully run your first for loop above "with d_test.values() iterable". 

• When I wrap d_test .values() with list() it works and prints my python environment info which matches yours. So now I need to figure out what is going on with my setup that is different from yours.

[HaydenWelch]

You're correct about the slight difference of list() loading the whole iterator (in this case a generator object for the dictionary values) into memory while the raw .values() call returns a generator with no memory allocation. My statement was more meant for this specific purpose that shouldn't make a difference. That is something that changed between python 2 and 3 though, so something internal to Arc Pro (your error is coming from ESRI's python parser, not Python itself) is raising that error.

It appears that the internal validation is being preformed by Analyze Tools For Pro  which uses the 2to3 library  And I'm actually getting the same errors as you for the toolbox when I run that on my toolbox:

Analyzing toolbox toolbox(C:<USERPATH>\Toolbox.pyt) for use within ArcGISPro:

WARNING 001683: Found Python 2 to 3 errors:

Line 57:         for i in d_test.values(): ->for i in list(d_test.values()):

Line 62:         for i in self.d_test.values(): ->for i in list(self.d_test.values()):

Line 77:         for i in iter(d_test.values()): ->for i in iter(list(d_test.values())):

Line 82:         for i in iter(self.d_test.values()): ->for i in iter(list(self.d_test.values())):

This is the rule that is triggered with this code:

I'm not sure why ESRI is reading this as an error though, because it's totally valid to iterate through a generator object. I'm assuming that whatever is happening is assuming that your toolbox is being ported from Python 2 and this specific rule exists to make Python 3's dict.values() -> generator behave like Python 2's dict.items() -> list

[gis_bCapell]

@HaydenWelch ,

I have had a coworker test and get the same results as you; list() isn't required when calling dict.values(). So, I am now trying to figure out what is wrong with my environment is throwing this error. 
When installing ArcGIS Pro 3.3 did you install for your user only or for anyone accessing the machine? Did you install the optional AI features? I am testing an uninstall/ re-install at the moment (for all users, no optional AI features).

[HaydenWelch · Accepted solution]

It's this setting, I don't have the issue because it was disabled by default in my previous User config which was ported to 3.3, but they must be enabling it by default now and that's insane... (Under Geoprocessing BTW)

Also please mark this reply as the solution. If this is a new thing they're doing the 3.3, then this issue is gonna start appearing more and I want people to be able to find the solution without reading through this whole thread lol