What’s New With Python in ArcGIS Pro 2.8 (The Inside Scoop!)

3967
10
05-26-2021 03:11 PM

What’s New With Python in ArcGIS Pro 2.8 (The Inside Scoop!)

We – the Python team at esri – would like to share what we think are the highlights of the ArcGIS Pro 2.8 release for our Python users.

 

Conda Upgrade

 

After long last (the last major conda upgrade was way back in ArcGIS Pro 2.1), conda has been upgraded from version 4.3.27 to version 4.9.2. That upgrade yields some excellent benefits. You can read through the conda release notes on your own, but here are our highlights:

  • The performance of conda has been substantially improved with the introduction of the .condafile format for distributing packages, so cloning environments and installing or upgrading packages have seen a significant speed boost when this format is available.

  • There are some new options with conda commands, we’d like to highlight some that we think are notable:

    • conda’s enhanced package query language, MatchSpec, can now bind individual packages to a specific channel using the :: operator. For example, say you would like to create a new Python environment from scratch and install both arcpy and pytorch. However, you don’t want to use pytorch from the esri channel, you specifically want to install pytorch version 1.7.1 from the pytorch channel. You can do this with the following command:

      conda create -n my_env python arcpy pytorch::pytorch==1.7.1


      MatchSpec can be used with create, install, update, remove, and search commands.

      On a side note – as of Pro 2.7, arcpy is a conda package. This means that it is now possible to create a valid environment from scratch as shown in the example above! You will, however, still need a licensed installation of ArcGIS Pro to be able to do this.

    • When you wish to install packages only from channels you designate, and not from the default channel or those specified in .condarc, you can now use --override-channels.

    • You can now prevent changing already-installed dependencies using the --freeze-installed flag on the install and update commands.

    • The new flag --envs added to conda search allows searching across all environments to find the environments that satisfy the search criteria (i.e., contain a particular package).

    • ArcGIS Pro uses an isolated conda installation and doesn't support conda init at this time.

Independent of the conda upgrade, installing geopandas into the same environment as arcpy is no longer a problem. Try this:

conda create -n my_env python arcpy geopandas

 

Notebooks in Pro

 

In the realm of notebooks in Pro, you can now choose the position of your notebooks when first opened in a Project. You can find this option in the Project Settings >> Option dialog under the User Interface tab. At the bottom, expand the Notebook section and choose from 1) stacking the map on top of the active map (default), docking the view below the active map, or opening the view in a separate window.

HannesZiegler_0-1622066506941.png

There’s also a new Export button on the Notebook ribbon, from which you can choose between 2) exporting the active notebook as a Python (.py) or HTML (.html) file.

HannesZiegler_1-1622066533849.png

 

Everything else that's new with Python & arcpy

 

Python has been upgraded from 3.7.9 to 3.7.10 to keep up with security patches. Nothing too exiting here, but there are also a few enhancements to arcpy to check out:

  • FeatureSet and RecordSet objects can now be created from Esri JSON format.

  • The SetLogMetadata and GetLogMetadata functions have been added to control whether dataset metadata is updated with geoprocessing tool information.

  • Geometry classes (Multipoint, PointGeometry, Polygon, and Polyline) now support named arguments.

  • The Array class can accept any iterable object that returns Point objects.

  • In the arcpy.mp (Mapping) module, the LayerTime class has been added to provide information about how time is stored and configured in time-enabled layers, and the MapTime class has been added to provide access to time management operations when time-enabled layers are displayed in a map frame on a layout.

  • In the arcpy.na (Network Analyst) module, solver result objects now have a spatialReference property and an extent method.

  • In the arcpy.sa (Spatial Analyst) module, the Aggregate and Slope geoprocessing tools now create a function raster output.

 

 

Comments

SetLogMetadata is a big one for us. We have many scripts running as scheduled tasks doing maintenance work every night. The metadata for these tasks was building up and slowing everything down. We now do arcpy.SetLogMetadata(False) for everything running as a scheduled task.

@BlakeTerhune great to hear you are finding this useful. This was implemented in response to requests by several of our customers, so it seems you are not the only one - or perhaps you were one of the customers that raised this to our attention. Thank you for the feedback!

Is there an obvious reason why my modules are no longer found while working in a Python Pro environment after the update to 2.8? My environment hasn't moved from this location:C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3

 

 

Will the arcgispro-py3 env (and clones) work in powershell or will we still need to keep using a separate miniconda installation that supports conda init to activate envs in powershell (and Visual Studio Code)? 

@Luke_Pinner Correct on the latter, for now conda init is disabled, so you will need to maintain a separate installation of conda to work in PowerShell. There has been some discussion around this internally, if you'd like to prod us further to get this implemented you can gain support for it by submitting an idea at ArcGIS Pro Ideas.

Is there an obvious reason why my modules are no longer found while working in a Python Pro environment after the update to 2.8? My environment hasn't moved from this location:C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3

@JaredPilbeam2  It's possible arcgispro-py3 was not properly removed. Before trying to fix that, did you install additional packages directly into C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3 (not advised)?

Also, if you are using a cloned environment that was created in a previous version of Pro, it will be a version mismatch issue. It's best to clone arcgispro-py3 and install the packages again if you know which you need and there aren't too many, or follow these steps:

To create a new environment that is up to date with the current ArcGIS Pro installation's arcgispro-py3:

  1. Generate a list of additional packages installed in your current environment,
    with the conda command: 
    >>> conda env export > env.yaml
  2. (Optional) If you have additional dependencies installed through pip,
    find those with pip freeze:
    >>> pip freeze > requirements.txt
  3. Create a new environment by cloning arcgispro-py3, and activate it:
    >>> conda create --clone arcgispro-py3 --name my-env --pinned
    >>> activate my-env
  4. Add back missing conda packages from your current environment:
    >>> conda env update -n my-env -f env.yaml
  5. (Optional) Add back missing packages from pip:
    >>> pip install -r requirements.txt
  6. (Optional) Make it the default for the ArcGIS Pro application and the
    "Python Command Prompt":
    >>> proswap my-env

@HannesZieglerThanks for the feedback. I never used the Pro Python Package Manager to install any packages. The Install/Uninstall buttons are always greyed-out anyway? And I've never set up a cloned environment. I didn't know what else to do but un-install Pro 2.8 and then reinstall 2.7. With 2.7 everything is functioning again.

With 2.8 I couldn't access anything python-related on top of not having the modules available in a stand-alone. Opening a new Notebook or the python window would crash Pro 2.8. And opening Jupyter from the ArcGIS folder in the Start menu also failed.

With this reinstall of 2.7 I'm still seeing duplicates in the Package Manager as I did with 2.8. I'm not sure how to uninstall when the button is greyed-out.

m.png

@JaredPilbeam2 I see, the trouble was likely caused by modifying arcgispro-py3. On the backstage, we try to discourage this by disabling uninstall/update/install packages (as you've noticed). The reason we discourage this is because if something breaks the default environment, Pro functionality can stop working. Note the orange warning message in the screenshot above: "Cannot modify the default Python environment (arcgispro-py3), Clone then activate a new environment first."

You should always clone you default environment. You can do this using the Manage Environments dialog and clicking the Clone Default button (cloning will usually take 1-2 minutes). Once you have your clone, you need to activate it from the list (select the radio toggle) and then restart Pro for this to take effect.

In your case, you'll need to take it a bit further to get back the packages you may have installed into arcgispro-py3 in your clone after you have cleaned up, I advise you take these steps:

  1. Navigate to C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3, back it up somewhere else, ex: %localappdata%\temp\arcgispro-py3-broken-backup
  2. Uninstall Pro 2.7
  3. Navigate to C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3 and make sure the folder is delted. If it isn't, go ahead and delete it. 
  4. Install Pro 2.8
  5. Now follow instructions 1-6 from my previous response using your backup environment arcgispro-py3-broken-backup to make sure you get all your installed packages in your new clone.

 

Learned some things. Thank you very much. When I uninstalled 2.8 the first time I didn't check whether the arcgispro-3 folder was there before I reinstalled 2.7. Looking back, it might have been due to the duplicate packages.

I didn't know it was a golden rule to work from a cloned environment. I'm pretty sure I've added packages to the arcgispro-3 environment in the past via the python command prompt or the VS Code 2019 package installer.

Everything is working as it should again.

@JaredPilbeam2 Good to know it's back to working, glad I could help! And thank you for letting us know we need to do a better job communicating that cloning the default environment arcgispro-py3 is strongly recommended, perhaps we need to work on our documentation to make sure it's harder to miss this detail and better explained.

@HannesZiegler @JaredPilbeam2 @BlakeTerhune I commented on a recent post (ref) and discuss some downsides of "conda init" and propose that ESRI  implement  better support for launching a powershell session that is aware of the conda that ships with ArcGIS Pro

Version history
Last update:
‎05-28-2021 08:06 AM
Updated by:
Contributors