topic Re: Execute Python 2 Code via Python 3 in Python Questions

Execute Python 2 Code via Python 3

WilliamCraft — Thu, 26 Dec 2019 19:58:51 GMT

I am trying to wrap some Python 2 code into Python 3 but having trouble getting the process to work correctly. From ArcGIS Pro, I'm running a custom GP tool with the following code as its script. My goal is to run my Python 2 code (stored in a separate script) against a series of MXDs in order to list the data source for each layer or table. In my case, doing it this way yields no results. The code runs in about 2 seconds and then terminates. Any ideas?

PYTHON 3 SCRIPT (WRAPPER)

import subprocess, os, winreg, sys, arcpy

try:
   arcpy.AddMessage("Finding Python 2.7 installation directory...")
   hKey = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE,    "SOFTWARE\\WOW6432Node\\Python\\PythonCore\\2.7\\InstallPath")
   result = winreg.QueryValueEx(hKey, "")[0]
except:
   arcpy.AddWarning("Python 2.7 installation directory was not found.")
   arcpy.AddError("Script failed.")
   sys.exit()

if os.path.exists(result + "\\python.exe"):
   arcpy.AddMessage("Launching Python 2.7 executable...")
   CREATE_NO_WINDOW = 0x8000000
   process = subprocess.Popen([result + "\\python.exe", "C:\\temp\\python\\ListMXDDataSources.py"],    stdout=subprocess.PIPE, stderr=subprocess.PIPE, creationflags = CREATE_NO_WINDOW, shell=True, stdin=None)
   stdout, stderr = process.communicate()
   arcpy.AddMessage('{}'.format(stdout.decode("utf-8")))
   arcpy.AddWarning('{}'.format(stderr.decode("utf-8")))

PYTHON 2 SCRIPT

import arcpy, glob

rootDirectory = 'C:/temp'
fileExtension = '.mxd'

def main():
   for f in glob.glob(rootDirectory + '/*' + fileExtension):
      mxd = arcpy.mapping.MapDocument(f)
      for df in arcpy.mapping.ListDataFrames(mxd, ''):
         for lyr in arcpy.mapping.ListLayers(mxd, '', df):
            if not lyr.isGroupLayer:
               if lyr.isRasterLayer:
                  print "Raster Layer, {}".format(lyr)
               elif lyr.isFeatureLayer:
                  print "Feature Layer, {}, {}".format(lyr, lyr.serviceProperties)
               else:
                  print "Layer, {}".format(lyr)
         for tbl in arcpy.mapping.ListTableViews(mxd, '', df):
            print "Table, {}".format(tbl.dataSource)

if __name__ == '__main__':
main()

Re: Execute Python 2 Code via Python 3

JoeBorgione — Sat, 11 Dec 2021 19:24:28 GMT

I've taken the liberty of copying and pasting your code sample into the syntax highlighter; you'll get more responses in this format.

Sorry , don't have any suggestions for you...

import subprocess, os, winreg, sys, arcpy

 

try:
   arcpy.AddMessage("Finding Python 2.7 installation directory...")
   hKey = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE,    "SOFTWARE\\WOW6432Node\\Python\\PythonCore\\2.7\\InstallPath")
   result = winreg.QueryValueEx(hKey, "")[0]
except:
   arcpy.AddWarning("Python 2.7 installation directory was not found.")
   arcpy.AddError("Script failed.")
   sys.exit()

 

if os.path.exists(result + "\\python.exe"):
   arcpy.AddMessage("Launching Python 2.7 executable...")
   CREATE_NO_WINDOW = 0x8000000
   process = subprocess.Popen([result + "\\python.exe", "C:\\temp\\python\\ListMXDDataSources.py"],    stdout=subprocess.PIPE, stderr=subprocess.PIPE, creationflags = CREATE_NO_WINDOW, shell=True, stdin=None)
   stdout, stderr = process.communicate()
   arcpy.AddMessage('{}'.format(stdout.decode("utf-8")))
   arcpy.AddWarning('{}'.format(stderr.decode("utf-8")))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Execute Python 2 Code via Python 3

WilliamCraft — Fri, 27 Dec 2019 15:03:41 GMT

Thanks Joe. The syntax highlighter isn't working properly for me so I was unable to do this myself. Also FYI, I've just now made some revisions to my code as well.

Re: Execute Python 2 Code via Python 3

StephanieWendel — Sat, 11 Dec 2021 19:24:31 GMT

Hey William Craft! Have you tried running the python 2 portions of the script in ArcMap's python window or in IDLE that comes with the Python2.7 install with Desktop? That could help you determine if it is a problem with python 3 or python 2.7. The other thing you could do is add a simplistic logging system within the script to record where things are failing. This always helps me narrow down if it is my code or the setup I'm running the script in. Below is my favorite logging system to add to scripts which I've added to yours as an example. Note: You could instead of using the script file location in the create_log function hard code a path in the location variable.

import subprocess, os, winreg, arcpy
import os.path, sys
from time import localtime, strftime

def create_log(name, headers=None, ftype=".txt", subfolder=None):
    location = sys.path[0]
    if subfolder is not None:
        location = os.path.join(sys.path[0], subfolder)
    time_setup = strftime("%m_%d_%Y", localtime())
    logfile = os.path.join(location, name + "_" + time_setup + ftype)
    if not os.path.exists(logfile):
        f = open(logfile, 'wb')
        if headers is not None:
            f.write(headers)
            f.write("\n")
        f.close()
    return logfile


def Log(logfile, message):
    f = open(logfile, 'ab')
    f.write(message)
    f.write("\n")
    f.close()

def main():
    logfile = create_log("Monitor", subfolder="Monitor_Logs")
    Log(logfile, "{} Created Log.\n".format(strftime("%m_%d_%Y_%H:%M:%S", localtime())))

    try:
        arcpy.AddMessage("Finding Python 2.7 installation directory...")
        Log(logfile, "Finding python 2.7 installation directory")
        hKey = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE,  "SOFTWARE\\WOW6432Node\\Python\\PythonCore\\2.7\\InstallPath")
        result = winreg.QueryValueEx(hKey, "")[0]
    except:
        arcpy.AddWarning("Python 2.7 installation directory was not found.")
        Log(logfile, "Python 2.7 installation directory was not found.")
        arcpy.AddError("Script failed.")
        sys.exit()

    if os.path.exists(result + "\\python.exe"):
        arcpy.AddMessage("Launching Python 2.7 executable...")
        Log(logfile, "Launching Python 2.7 executable...")
        CREATE_NO_WINDOW = 0x8000000
        process = subprocess.Popen([result + "\\python.exe", "C:\\temp\\python\\ListMXDDataSources.py"],    stdout=subprocess.PIPE, stderr=subprocess.PIPE, creationflags = CREATE_NO_WINDOW, shell=True, stdin=None)
        stdout, stderr = process.communicate()
        arcpy.AddMessage('{}'.format(stdout.decode("utf-8")))
        Log(logfile,'{}'.format(stdout.decode("utf-8")))
        arcpy.AddWarning('{}'.format(stderr.decode("utf-8"))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Execute Python 2 Code via Python 3

WilliamCraft — Sat, 28 Dec 2019 01:06:00 GMT

Thank you Stephanie Wendel for your help. The failure seems to be associated with the site.py file of Python 3. The Python 2 script works perfectly fine on its own. Here is the underlying error after printing the stderr output from the subprocess.Popen line in the Python 3 code (line 45 above):

Re: Execute Python 2 Code via Python 3

JoshuaBixby — Sat, 28 Dec 2019 22:58:54 GMT

The screenshot is helpful. The answer to your question/issue is that an ArcGIS Pro Python module is being called with a Python 2.7 interpreter. In line 177 of site.py, the code uses a syntax for print that is only valid in Python 3 and not Python 2, hence the invalid syntax error.

Re: Execute Python 2 Code via Python 3

DanPatterson_Retired — Sat, 11 Dec 2021 19:24:34 GMT

from __future__ import print_function
print("hello")  # not   .... print "hello"‍‍

if you want to fix up the 2.7 script

Re: Execute Python 2 Code via Python 3

curtvprice — Sun, 29 Dec 2019 01:45:54 GMT

Dan, I don't think this will make any difference. Note that in 2.7 print() works just fine without the from __future__ import... I always use it now in my ArcMap code so my scripts will run in both - of course the OPs script can't run in both because is uses ArcMap arcpy methods.

William, it looks to me from your error message that the PATH environment is causing the Pro site.py to get executed when Python 2.7 starts up. The way to fix this is to make sure the shell you start with Popen has a PATH environment that doesn't include the Pro folders. This is done using the env parameter to Popen.

Re: Execute Python 2 Code via Python 3

WilliamCraft — Mon, 30 Dec 2019 15:15:13 GMT

Thank you everyone for your help. Curtis, the suggestion you made seemed to get me past the error I was experiencing. I set the PATH, PYTHONHOME, and PYTHONPATH variables to only reference the Python 2.7 folder (C:\Python27\ArcGIS10.6). However, I'm now getting a new error when the 2.7 code executes. Note that that 'result' variable is equivalent to the value of "C:\Python27\ArcGIS10.6".

process = subprocess.Popen(result + "\\python.exe C:\\temp\\python\\ListMXDDataSources.py", stdout=subprocess.PIPE, stderr=subprocess.PIPE, creationflags = CREATE_NO_WINDOW, shell=True, env={'PATH': result, 'PYTHONHOME': result, 'PYTHONPATH': result})

Re: Execute Python 2 Code via Python 3

Arne_Gelfert — Sat, 11 Dec 2021 19:24:37 GMT

Okay, i don't typically partake in troubleshooting discussions this deep below the surface (and yet so way above my head). But this thread is producing some "keeper" bits of information.

Looks like your code fails at the use of seed() at _hexlify(). Never heard of it but when you compare random.py in 2.7 and 3.6 (my machine), you notice that hexlify shows up in 2.7...

from __future__ import division
from warnings import warn as _warn
from types import MethodType as _MethodType, BuiltinMethodType as _BuiltinMethodType
from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
from math import sqrt as _sqrt, acos as _acos, cos as _cos, sin as _sin
from os import urandom as _urandom
# =======================================
from binascii import hexlify as _hexlify
# =======================================
import hashlib as _hashlib‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

but not in 3.6

from warnings import warn as _warn
from types import MethodType as _MethodType, BuiltinMethodType as _BuiltinMethodType
from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
from math import sqrt as _sqrt, acos as _acos, cos as _cos, sin as _sin
from os import urandom as _urandom
from _collections_abc import Set as _Set, Sequence as _Sequence
from hashlib import sha512 as _sha512
import itertools as _itertools
import bisect as _bisect
‍‍‍‍‍‍‍‍‍

So maybe you still have 2.x code searching for something in 3.x environment?

Re: Execute Python 2 Code via Python 3

WilliamCraft — Mon, 30 Dec 2019 16:30:14 GMT

I think I figured out how to get past this issue too, thanks to your comment, the information from Curtis and Joshua, and some online investigation into the error code from the screenshot above. I added the following bold text to the environment parameter of my popen line:

Below is where things currently stand when I run my code as of now. I think I'm getting much closer now. Thank you to everyone.

Re: Execute Python 2 Code via Python 3

JoshuaBixby — Sat, 11 Dec 2021 19:24:39 GMT

William Craft‌, how exactly are you executing your Python 3 script? I just took the original code you posted when you first opened this thread and it runs fine for me with some test MXDs:

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>python D:\tmp\ListWrapper.py
Finding Python 2.7 installation directory...
Launching Python 2.7 executable...
Layer, wwf_terr_biome_type_smooth

WARNING:

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>‍‍‍‍‍‍‍‍

Re: Execute Python 2 Code via Python 3

WilliamCraft — Mon, 30 Dec 2019 17:48:54 GMT

I'm running it from ArcGIS Pro's catalog view via a custom toolbox/toolset. There seems to be an issue with referencing the map document file when creating the MXD object. Are you running the code within the Python window in ArcGIS Pro?

Re: Execute Python 2 Code via Python 3

JoshuaBixby — Mon, 30 Dec 2019 17:52:10 GMT

I ran the code from the "Python Command Prompt" shortcut that is created when Pro is installed. That shortcut just takes care of activating the correct/default Pro Python environment. If you run your code from that command prompt, does it generate an error?

Re: Execute Python 2 Code via Python 3

WilliamCraft — Mon, 30 Dec 2019 17:57:29 GMT

Doing it the way you suggested still yields the same error for me: "RuntimeError: Object: CreateObject cannot open map document". Looks like maybe our environments are different enough where it'll run for you but not me. I am curious why my environment prevents the MXD object from being created. Thanks for looking into this a bit more. Much appreciated.

Re: Execute Python 2 Code via Python 3

JoshuaBixby — Mon, 30 Dec 2019 18:09:56 GMT

The RuntimeError you are now experiencing might be caused by a problematic MXD, not a Python 2/3 environment issue. Are you getting the RuntimeError with any/all MXDs? Can you try just a single MXD in a directory.

Re: Execute Python 2 Code via Python 3

WilliamCraft — Mon, 30 Dec 2019 18:36:38 GMT

I tried it with a newly-created, simple MXD containing a feature class from a FGDB and a basemap. I also tried this on another computer, both having the same version of ArcGIS Pro 2.4; one computer had ArcGIS Desktop 10.6.1 and the other had 10.7.1. Same result on both machines. I'm so confused.

Re: Execute Python 2 Code via Python 3

WilliamCraft — Mon, 30 Dec 2019 21:36:57 GMT

In the Python 2 script, I added an os.startfile(f) line of code to launch the 10.6 MXD interactively just to see what issues may arise. The following errors are found when taking this approach:

I have tried deleting my Normal.mxt but that only eliminates the first error shown above. I can open this map document just fine by double clicking on it from Windows Explorer.

Re: Execute Python 2 Code via Python 3

JoshKalov — Sat, 11 Dec 2021 19:24:42 GMT

You have already set the PATH in env so this may not be your issue, but I have gotten a python2 call from python3 to work by doing a os.environ.copy() first before updating the path.

replica_sync_command = f"C:\\Python27\\ArcGIS10.4\\python.exe c:\\syncReplica_python2.py"
python2_env = os.environ.copy()
python2_env.update({"PATH": "C:\\Python27\\ArcGIS10.4"})
run_sync = subprocess.run(replica_sync_command.split(), env=python2_env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Execute Python 2 Code via Python 3

JoshuaBixby — Tue, 31 Dec 2019 16:18:47 GMT

I was able to use os.startfile() from both a Python 2 (ArcMap Python) and Python 3 (Pro Python) command prompt to open multiple MXDs in ArcMap. I say this just to say it isn't an issue with os.startfile(), so resolving these errors might be a good step in resolving your larger issue.