Skip navigation
All Places > Developer Communities > GIS Developers > Python > Blog

Python

9 posts

IDLE is easy and fast and is always there with Python distributions. It isn't fancy, but it does get the job done. Old Unix hands will nod in agreement when it is praised as "the vi of Python IDEs".

 

The default installed Python shortcut in your start menu IDLE (Python GUI) opens one of the ArcGIS Pythons you have available (ArcMap 2.7 x32 or x64). As there can be as many as three (those two plus ArcGIS Pro's main Conda Python environment), you may want to test Python code using others.  

 

Here is how to do create desktop shortcuts to open IDLE with these other ArcGIS Python installations. These instructions assume you have both ArcMap and Pro installed.

 

Step 1. Create a shortcut on the desktop for IDLE

  1. Do a Windows search for IDLE  (type IDLE in the search box)
  2. Right click IDLE (Python GUI). This is the default shortcut installed with ArcMap. Select "Open File Location"
  3. The folder that opens contains the ArcMap Python shortcuts.

Step 2. Create an IDLE shortcut for ArcGIS Pro's Python environment

  1. Open the file explorer and navigate to: <Install Folder>\bin\Python\envs\arcgispro-py3\Lib\idlelib

  2. Right click the file idle.bat Send to > Desktop (create shortcut) 

  3. On the desktop find idle.bat - shortcut right click > Rename to a more useful name like IDLE Pro
  4. If you want open the shortcut's properties and assign an icon to make it easy to identify. I imported the icon from the file <Install Folder>\bin\Python\envs\arcgispro-py3\Lib\idlelib\icons\idle.ico

Step 3. Create an IDLE shortcut for ArcMap's x32 Python environment (optional)

If you have ArcMap x64 background geoprocessing installed, the IDLE (Python GUI) start menu item uses its Python (Python 2.7 x64) n. However, you may want to create an IDLE shortcut to access the Python 2.7 x32 that corresponds to ArcMap's native 32-bit Python environment ("foreground"). (If you don't have the x64 background processing installed, the default shortcut IDLE (Python  GUI) will already open Python 2.7 x32.)  You would want this shortcut to run scripts that only run in x32 geoprocessing, for example scripts that use the metadata GP tools or read personal (.mdb) geodatabases.

  1. On the desktop, copy and paste IDLE (Python GUI), this will create  IDLE (Python GUI) - Copy

  2. Rename IDLE (Python GUI) - Copy to IDLE ArcMap32

  3. Right click IDLE ArcMap32> Properties

  4. Modify Target string to remove x64 from the path:
    from: C:\Python27\ArcGISx6410.6\pythonw.exe "C:\Python27\ArcGISx6410.6\Lib\idlelib\idle.pyw"
    to: C:\Python27\ArcGIS10.6\pythonw.exe "C:\Python27\ArcGIS10.6\Lib\idlelib\idle.pyw"

 

References

Writing Python scripts—Help | ArcGIS Desktop 

Background Geoprocessing (64-bit)—Help | ArcGIS Desktop 

Python in ArcGIS Pro—ArcPy Get Started | ArcGIS Desktop 

Localized and silent install options—ArcGIS Pro | ArcGIS Desktop  (INSTALLDIR)

If you see this error message from a Python script tool (usually during an import):

ImportError: DLL load failed: %1 is not a valid Win32 application

it is likely you have some non-ArcGIS versions of Python installed that messed with the environment. The issue really is how weak Windows is with how it handles paths. The way most people get into this trouble is when they install an additional (non-ArcGIS) Python package (for example, Anaconda) and neglect to un-check the boxes that modify the Windows environment. This is a sure way to break the Windows setup (registry, PATH, file associations) ArcGIS applications need to properly communicate with the correct Python executables and run in the correct environment.

The fix is to go to Add/Remove programs and remove any Python installs you see there, then run a repair install of ArcGIS Desktop (and ArcGIS 64 bit Background Processing, and Pro, if that's installed too). This will get you on the true Python path.

 

Once you've done this, you can install other Pythons, but you have to be very careful not to modify the Windows environment when you do so. It can get pretty involved -- if you can I highly recommend sticking with the (now pretty deep) Python stack shipped with ArcGIS. Then the scripts you write will work with anyone with that version of ArcGIS installed without any additional Python installs. Sometimes this isn't possible, but when it is you are saving yourself and users of your scripts a lot of trouble by sticking with the ArcGIS as-shipped Python environment.

Using Anaconda modules from the ESRI python environment

Using a user-specific Python startup script

A Python addin enables you to create custom button, tool, combobox on your custom toolbar and create your own extension in Python language. Use Esri's desktop and online documentation - it has all the information to get started, create, debug and maintain Python addins.

 

However, I have seen users make mistakes that are hard to find or debug. So, I'm listing few guidelines which may save you time and in many cases get you out of trouble.

 

Project Settings

 

There is no restrictions on setting Project properties while creating an addin, however, having a good name for the project and an identifiable image as icon will help.

Python Addin Wizard - project settings.

 

Setting Toolbar properties

 

Do not use any space in setting toolbar ID.

Setting toolbar properties

Setting Extension properties

 

Same rule applies for naming the ID variable. However, I suggest naming the class such that, when coding the script-behind, you know that it is a class - also, class name cannot contain space. Additionally, initially select only those methods that you'd need in your extension. 

 

Setting extension properties

 

Setting Button, Tool, ComboBox properties

 

ID (Variable Name) is important - portion of the ID right to dot is the actual ID. This is very important as you use this ID to communicate with other controls (or extension) of the same add-in. The wizard gives generic button1, button2 names but when you write code for the controls you can be confused which does what - it will be easier if the ID itself contains clue.

Setting properties for Button, Tool and ComboBox

Using Global variables for communication between or among controls

 

It is not well-advised to use global variables in Python - same advise applies to writing code for Python addins. Instead, you can use an Extension class with your own methods to facilitate communication. Or, you can have your own class to do the same.

 

Please feel free to comment if you have suggestions for improvements of these suggestions.

Often we have scripts that require a home folder location for the script to locate related files -- for example, layer files, data distributed with the tool that it uses, etc.  Here are some methods to find that location.

 

os.path.dirname(sys.argv[0]) or os.path.dirname(__file__)

Both these locations find the path of the currently running Python script file.

Usually, __file__ is the best choice.
See: Difference between __file__ and sys.argv[0] (StackOverflow)

 

Toolbox (.tbx) validation code

In .tbx script tool validation code, sys.argv[0] returns an empty string, but  __file__ will be the path to the function packed in a string like this: u'C:\\test\\Toolbox.tbx#Script_z.InitializeParameters.py'

In this case, the location of the tbx would be: os.path.dirname(__file__)

 

ModelBuilder tbx location

The ModelBuilder tbx location can be found using os.path.realpath and the Calculate Value tool with this code block:

# Call with expression: home()
import os
def home():
  return os.path.realpath(".")

 

Map document location ("Home folder" in ArcMap or Pro)

Some script tools (always run from ArcMap or Pro) may need to locate data relative to the .mxd or .aprx location:

# ArcMap
here = os.path.dirname(arcpy.mapping.MapDocument("CURRENT").filePath)
# ArcGIS Pro
here = arcpy.mp.ArcGISProject("CURRENT").filePath

I have been attempting to get into Python multiprocessing. Just a warning, it's not for the faint of heart. It's great when it works, but there are many gotchas! Here's some background on that if you want to read up:

If your python script tool uses parallel processing, you may want your tool to recognize a new environment setting recently added to ArcGIS: parallelProcessingFactor:

Parallel Processing Factor (Environment setting)—Help | ArcGIS for Desktop

 

from the help article:

The value of this environment determines the number of processes across which a tool spreads its operation. Those processes will be divided between hardware cores (processors) built into the machine. The number of hardware cores does not change based on this setting.

Here's a function that will convert this setting to the number of cpus argument used for multiprocessing.Pool()

UPDATE: now supports 0 - this value can be used to signal your script that no multiprocessing should take place. Also I added better exception handling.

 

def arcpy_cpus(default=None):
    """Determine number of CPUs to use with Python multiprocessing.Pool()
    based on arcpy.env.parallelProcessingFactor.
    """
    import multiprocessing
    import arcpy
    ppf = arcpy.env.parallelProcessingFactor
    if ppf in ["", None]:
        if default in ["", None]:
            ppf = multiprocessing.cpu_count() - 1
        else:
            ppf = default
    if str(ppf) == "0":
        cpus = 0 # no multiprocessing
    try:
        if '%' in str(ppf):
            pct = float(ppf.replace("%",""))
            cpus = int(multiprocessing.cpu_count() *  (pct / 100.0))
        else:
            cpus = int(ppf)
            if cpus < 0:
                raise
    except Exception as msg:
        raise Exception("Invalid parallelProcessingFactor \"{}\"".format(ppf))
    return cpus

 

Here are some test runs of this function, so you can see what it returns with different parameters:

 

print("{:8} {:5}  {:5}".format("env", "def=\"\"", "def=2"))
for k in ["0", "1", "", None, "1%", "2%", "50%", "100%", "200%"]: 
   env.parallelProcessingFactor = k
   print ("{!r:8} {:5}  {:5}".format(k, arcpy_cpus(), arcpy_cpus(2)))


env      def=""  def=2
'0'          0      0
'1'          1      1
''           7      2
None         7      2
'1%'         0      0
'2%'         0      0
'50%'        4      4
'100%'       8      8
'200%'      16     16

The following script gives you the ability to fully customize your python environment.

 

The example below is set up for ArcGIS 10.2 with ArcGIS background GP and  anaconda 32 and anaconda 64 installed.

 

You can tweak this to your hearts content to adapt to all your favorite Python environments!

 

# Startup script to link Anaconda python environment with ArcGIS
#
# 1. Install Anaconda, setup environment to match your ArcGIS version
# 2. Edit the paths below
# 3. Put this startup script in the startup folder
#    Startup folder can be found with: "C:\Python27\ArcGIS10.2\python -m site --user-site"
#    Usually will be:
# C:\Users\%USERNAME%\AppData\Roaming\Python\Python27\site-packages


# edit these paths to match your setup


arcver = "10.2"
# Anaconda home folders
conda32 = r"D:\Users\cprice\Anaconda"
conda64 = r"D:\Users\cprice\Anaconda64"
# here are the conda environments you've set up use with ArcGIS
# arc1022 is the environment setup for ArcGIS
conda_env32 = "{}/envs/{}".format(conda32, "arc1022")
conda_env64 = "{}/envs/{}".format(conda64, "arc1022")


# do not edit below this line


# ArcGIS Python home folders
# i.e. C:\Python27\ArcGIS10.2
arcver = arcver[:4]
arcpy32 = r"C:\Python27\ArcGIS{}".format(arcver)
arcpy64 = r"C:\Python27\ArcGISx64{}".format(arcver)

import sys
import os

try:
  if sys.version.find("64 bit") < 0:
    conda_path = os.path.normpath(conda_env32)
    arcpy_path = os.path.normpath(arcpy32)
    arcpy_pthfile = arcpy_path + "/lib/site-packages/desktop{}.pth".format(arcver)
  else:
    conda_path = os.path.normpath(conda_env64)
    arcpy_path = os.path.normpath(arcpy64)
    arcpy_pthfile = arcpy_path + "/lib/site-packages/DTBGGP64.pth"


  # If running ArcGIS python, add conda modules to path
  if sys.prefix.lower().find("arcgis10") != -1:
    conda_site = os.path.join(conda_path, "lib", "site-packages")
    if not os.path.isdir(conda_site):
      raise Exception()
    sys.path.append(conda_site)
    ##print("added conda paths to arc")

  # if running Anaconda add arcpy to path
  elif sys.prefix.lower() == conda_path.lower():
    with open(arcpy_pthfile, "r") as f:
      sys.path +=  [p.strip() for p in f.readlines()]
    ##print("added arcpy paths to conda")

except Exception as msg:
  print(msg)
  pass
curtvprice

Fixing broken Python

Posted by curtvprice Champion Jan 27, 2016

Sometimes the Python environment can get broken so ArcGIS will not work right; import arcpy fails, or tools will not open. Not an uncommon problem. A full install of ArcGIS includes both x32 (ArcMap) and x64 (background GP) Python 2.x installations, and ArcGIS Pro if installed adds a third one (x64 Python 3.4).

 

The Python environment often is broken when a non-ArcGIS installation of Python has been installed and the system paths are altered. Although there are ways to manage multiple Python environments, the Windows file associations, etc must be kept clean for ArcGIS to access Python directly. If you do install a 3rd part Python, say, Anaconda, be sure NOT to check the boxes to modify the default Windows PATH environment and file associations.

 

If your Python setup is not working, and you having trouble resolving the issues, here is a workflow that should get ArcGIS Python functions working again:

 

1. Close ArcGIS

2. Make sure your PYTHONPATH is blank and PATH environment is clear of Python references (setting env variables)

3. Uninstall any Python and additions (PythonWin etc) you see in Add/Remove Programs

4. Make sure your ArcGIS installer media is available  (DVD or folder with the installer Setup.exe / Setup.msi files)

5. Rename Python folders on C drive

For example: rename C:\Python27 to C:\Python27_old) (renaming is just in case you have some files you want to copy over to the fixed python -- experts only!)

6. Repair Install ArcGIS

7. Repair Install ArcGIS 64-bit Background Processing

8. Repair install other ArcGIS software (Interoperability Extenson, Production Mapping, etc.)

9. If you want to download and install PythonWin matching your version of Python. (It's probably best to install x32 PythonWin unless you have a special need for x64.)

10. Reset your ArcGIS application profile

 

Completely uninstalling and reinstalling ArcGIS is major Windows surgery and should be a last resort.

I have run across the need to save and restore the arcpy environment. For example, a function (or a tool packaged in a function) may alter the environment and it would be nice to have an easy way to set it back at the end.

 

Arcpy provides the functions SaveSettings and LoadSettings, but they require that an XML file be created. I was having problems with these functions and had a epiphany that this could be more easily done by combining the ListEnvironments function and a Python dictionary.

 

Enjoy!

 

def SaveEnv():
    """Save selected arcpy environments as dict"""
    env_dict = {}
    for e in arcpy.ListEnvironments():
        if e not in ["scratchGDB", "scratchFolder", "packageWorkspace"]:
            env_dict[e] = arcpy.env[e]
    return env_dict

def LoadEnv(env_dict, display=False):
    """Load selected arcpy environments from dict"""
    for e in env_dict.keys():
        arcpy.env[e] = env_dict[e]
        if display:
            arcpy.AddMessage("{0:<30}: {1}".format(e, env_dict[e]))
            print("{0:<30}: {1}".format(e, env_dict[e]))

 

Here's how I am using these functions to keep GISProc from messing up environment when called:

 

def GISProc(arg1, arg2):
    envs = SaveEnv()
    try:
        # Tool goes here ...
        arcpy.env.extent = ...
        arcpy.env.cellSize = ...
    except:
        ...
    finally:
        LoadEnv(envs)

I have found this helpful by looking through some documentation on stack exchange a while back.  Thought that I might share this here as well.

 

This is a script that can be used to create a new MXD for ArcMap in python.  This will allow you to create a new MXD and then modify it using the ArcPy mapping module.  This functionality is not currently present in ArcPy so this script uses comtypes to call ArcObjects to create it.  Depending on what version of ArcMap that you are running, you may need to change line 9 or 10 in the Snippets.py file to point to the correct assembly reference.  (You will want to run run.py)

 

Modify line 8 in run.py to specify the location of the MXD to be created.

 

Sources used:

Accessing ArcObjects from Python?

http://gis.stackexchange.com/questions/80/accessing-arcobjects-from-python

 

Using ArcObjects in Python

http://www.pierssen.com/arcgis/upload/misc/python_arcobjects.pdf

 

I hope some users will find this useful.