topic Re: Distributing virtual environment to coworkers in Python Questions

Distributing virtual environment to coworkers

Glasnoct — Fri, 10 Jun 2022 19:33:25 GMT

Our current toolbox of scripts requires the installation of some modules that do not come with ArcPro out of the box and the current solution has been to have coworkers fire up the Python command prompt and do a "pip install" for the missing modules but that is proving to be too advanced for them and thus I'm looking for a simpler solution.

What's the easiest way I can keep everyone up to date with any dependencies we need? Can I just do a copy-paste of the virtual environment folder? Could I clone the environment to a shared directory and have them point their project environment to that? That would be useful as I can update it as necessary and it will "populate" to all the other users.

Re: Distributing virtual environment to coworkers

Anonymous User — Sat, 11 Jun 2022 12:41:55 GMT

That is timely... I am working the same process for a department. I'll share what I did, but I am also very interested in seeing what others have done as well. I haven't really found a solid solution of combining/ deploying custom python environments along with add-ins, which seems they would go hand in hand. If you are making an add-in to run a custom script... most likely you'll have a cloned environment and packages that go along with it right? It would be awesome if esri could provide a means to create/ manage environments through the SDK.

I dread that when Pro updates, it will constantly invalidate the new environment. With this in mind, I need a way that the user can recreate the environment each time, without me having to touch each machine after each update. I also need to be able to capture packages that they might have imported for other scripts, so they don't have to constantly switch environments or have to re-install each time.

I came up with a bat file script that I run from an add-in button (it can be ran as a bat file from the desktop as well) to clone the pro default env, create a yml from a cloned env, install the missing packages, copy local in-house packages, and sets the active environment in Pro to the new env. I've had some issues with it- jupyer-notebook seems to be a pita and fails to install sometimes, which causes the transaction to rollback. numpy sometimes complains that it wasn't installed correctly on 1/4 of the pc's I've ran this on.

I'm at the stage of testing deployment and adjusting commands to see if I can get it working 100%, so the -freeze-installed --no-update-deps is trial to hopefully fix this. I'm not a conda pro by any means, so if there is a better command I should be using, please feel free to let me know. I tried to comment the code to explain each step and when I have more time, I'd like to incorporate if statements for using the clone-1 environment if it exists but for now, this gets me 98% of the way there very quickly. You can grab the yml and save it where it can be accessed and install from there too by reference the path in the conda env update line.

I saved a requirements.txt that has a list of additional packages I need, (grabbed from the end of the yml) and do a second pip for these to make sure that they get installed.

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: VARIABLES :
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
@echo off

SETLOCAL

REM set environment names
SET "ENV_NAME=assessors-env"
SET "ARCGIS_DEFAULT=arcgispro-py3"
SET "ARCGIS_CLONE=arcgispro-py3-clone"

REM Paths set to variables
SET "PRO_PATH=C:\Program Files\ArcGIS\Pro\bin\Python"
SET "CLONED_PATH=%LOCALAPPDATA%\ESRI\conda\envs"
SET "REQ_PATH=\\gisdata\Assessor\ASR_ArcGIS Pro"

REM Set the default scripts dir path for this iteration for pip installing
SET PATH=%PATH%;"C:\Program Files\ArcGIS\Pro\bin\Python\Scripts"
REM Set the new env scripts dir path for this iteration for pip installing
SET PATH=%PATH%;"%PRO_PATH%\envs\%ENV_NAME%\Scripts"

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: COMMANDS :
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

REM start by activating the arcgis pro conda env. If successful, continue process
call "C:\Program Files\ArcGIS\Pro\bin\Python\Scripts\activate.bat" & (
REM export the cloned arcgispro-py3-clone environment that contains the additional packages installed outside of arcgis pro
conda env export -n %ARCGIS_CLONE% > "%CLONED_PATH%\%ARCGIS_CLONE%\environment.yml"
ECHO ^-^-^> %ARCGIS_CLONE% conda environment exported to "%CLONED_PATH%\%ARCGIS_CLONE%\environment.yml"

REM clone the deafult arcgispro-py3 environment to the new environment name. If needed, you can add specific package channels
REM CALL conda config --add channels conda-forge
REM CALL conda config --add channels esri
conda create --clone %ARCGIS_DEFAULT% --name %ENV_NAME% --pinned
ECHO ^-^-^> %ENV_NAME% environment created

REM install packages that were not installed from the default clone, but were in the arcgispro-py3-clone environment.
REM conda env list
conda env update -n %ENV_NAME% -f "%CLONED_PATH%\%ARCGIS_CLONE%\environment.yml --freeze-installed --no-update-deps"
ECHO ^-^-^> %ENV_NAME% environment updated

REM ensure that the new env is updated with the pip installed packages at the end of the environment.yml file.
pip install -r "%REQ_PATH%\Support Scripts\requirements.txt"
ECHO ^-^-^> %ENV_NAME% environment updated pip

REM copy local packages to the env
robocopy "%REQ_PATH%\cntytools" "%PRO_PATH%\envs\%ENV_NAME%\Lib\site-packages\cntytools" /E

REM set the active environment in Pro to the new environment
CALL "%PRO_PATH%\Scripts\proswap.bat" %ENV_NAME%
ECHO ^-^-^> Pro env set
exit
)

exit

Re: Distributing virtual environment to coworkers

Anonymous User — Sat, 11 Jun 2022 13:03:08 GMT

This is the daml for the add-in button. If you look at the Ribbons control solution in Pro SDK github repo, you'll see how to add ribbons and buttons to an add in.

<tabs> <tab id="RibbonControls_Tab1" caption="Admin Tools"> <group refID="RibbonControls_PythonEnv" /> </tab> </tabs> <groups>  <group id="RibbonControls_PythonEnv" caption="Python Env">  <button refID="RibbonControls_PythonEnv_Btn" size="large"/>   </group> </groups> <controls>   <button id="RibbonControls_PythonEnv_Btn" caption="Python Environment" className="PythonEnv_Btn" loadOnClick="true" smallImage="Images\Python-PNG-Clipart.png" largeImage="Images\Python-PNG-Clipart.png"> <tooltip heading="Establish Assessors-env python env"> Creates the assessors-env python environment and installs the required packages.<disabledText /> </tooltip> </button> </controls>

This is the button logic for the add-in to run the bat file. The bat file is added as content, always copied as the pro snippets documentation for files suggest.

internal class PythonEnv_Btn : Button { protected async override void OnClick() { var batFile = Path.Combine(Path.GetDirectoryName(Uri.UnescapeDataString(new Uri(Assembly.GetExecutingAssembly().CodeBase).LocalPath)), @"Utils\python-env.bat"); await QueuedTask.Run(async () => { try { ProcessStartInfo processInfo = new ProcessStartInfo { FileName = batFile, UseShellExecute = false, CreateNoWindow = false }; processInfo.UseShellExecute = false; processInfo.RedirectStandardError = false; processInfo.RedirectStandardOutput = false; using (Process process = Process.Start(processInfo)) { process.WaitForExit(); process.Close(); } // Show a messagebox with the results MessageBox.Show(@"Check if assessors-env was created at C:\Program Files\ArcGIS\Pro\bin\Python\envs."); } catch (Exception exc) { // Catch any exception found and display in a message box MessageBox.Show($"Exception caught while trying to validate the python env: {exc.Message}."); return; } }); } }

It opens a cmd and runs the bat file. When you see (env name) C:> stuff/stuff/stuff> , you can close the window. It doesn't take any more instructions from the bat file after it activates the new environment and I haven't found a way to close it yet.

Re: Distributing virtual environment to coworkers

Anonymous User — Sat, 11 Jun 2022 13:33:33 GMT

Ok, last one but I just thought of it after submitting the last post. You could temporary set the paths to a python package library location at the beginning of your scripts to point to the additional packages you need.

os.environ["PATH"] = 'path to your packages'

Re: Distributing virtual environment to coworkers

Glasnoct — Mon, 13 Jun 2022 15:59:56 GMT

Lordy that's a lot of info! My use case doesn't involve any add-ins since that's a whole can of worms I doubt I'll ever open. It looks like the .bat file you have there also involves cloning the default environment. My coworkers already have cloned environments so I'm thinking the easiest way forward would just be to have them run a batch file that tries to install all the required modules. This way I can update the batch file when I have new requirements and all I need is them to run it (or even better, maybe it could run as part of the tool script? Could I do that?)

What's the barebones batch file I can distribute to them that'll just do a pip install of a module list via their currently active environment?

Re: Distributing virtual environment to coworkers

Anonymous User — Mon, 13 Jun 2022 18:41:01 GMT

True, It’s a lot because it has to do a lot so I do t have to do a lot. 🙂

Ill be away for a while without internet but barebones is these two lines- replace the %variable% with your path to the requirements.txt file.

Files\ArcGIS\Pro\bin\Python\Scripts\activate.bat" & (
REM ensure that the new env is updated with the pip installed packages at the end of the environment.yml file.
pip install -r "%REQ_PATH%\Support Scripts\requirements.txt"
ECHO ^-^-^> %ENV_NAME% environment updated pip)

Re: Distributing virtual environment to coworkers

Glasnoct — Mon, 13 Jun 2022 19:35:44 GMT

Did a bit of your code get chopped off there? %REQ_PATH% isn't declared prior to usage (or doesn't need to? I don't really .bat) and it starts off with "Files".

edit: I gotcha; I just realized it's snipped from the original batch file. I'll piece it together.

edit2: Nope, played with it a bit but can't get it to work. activate.bat complains about "pip" and if I try "python pip" instead it complains that python can't be found. Trying it with proenv.bat activates fine but the command doesn't pass through and nothing on Google is answering how to pass the pip command after calling proenv.bat

Re: Distributing virtual environment to coworkers

Glasnoct — Wed, 15 Jun 2022 18:20:51 GMT

For future reference this is what I arrived at finally. stick it in a .py and import it as a module into your script. Edit the required variable to include the names of any modules that will need to be pip installed if not already. No .bat files required and installs to the user's current active conda environment.

import arcpy import sys import subprocess import pkg_resources required = {'excel2img', 'pypdf2', 'ahk', 'ahk-binary', 'pysimplegui'} def check_required_modules(mod_set: set)-> None: """ Compares input set against current venv installed modules and installs them if missing e.g. mod_list = {'excel2img', 'pypdf2', 'ahk', 'ahk-binary', 'pysimplegui'} :param mod_set: string set of module names to check :return: """ installed = {pkg.key for pkg in pkg_resources.working_set} missing = mod_set - installed if missing: python = r"c:\Progra~1\ArcGIS\Pro\bin\Python\scripts\propy.bat" for required_module in missing: subprocess.check_output([python, '-m', 'pip', 'install', required_module], shell=True) msg = f"{required_module} was missing and was installed" print(msg) arcpy.AddMessage(msg) else: msg = 'all required modules are installed! :)' print(msg) arcpy.AddMessage(msg) sys.exit() check_required_modules(required)

Re: Distributing virtual environment to coworkers

Anonymous User — Fri, 24 Jun 2022 13:12:45 GMT

That is a good solution, glad you were able to get something figured out!

Re: Distributing virtual environment to coworkers

Anonymous User — Fri, 24 Jun 2022 13:31:13 GMT

I was posting through my phone and had to keep it simple. Geonet isn't very mobile friendly and I am glad that it made enough sense for you to at least try it, but I don't think I copied enough for it to work.

I came into that issue too with pip, activate.bat and proenv.bat as well. To get around it (and what I didn't grab to paste in the last post), I set a temp environment variable pointing to where the scripts directory is for pip.exe:

SET PATH=%PATH%;"C:\Program Files\ArcGIS\Pro\bin\Python\Scripts"

SET PATH=%PATH%;"%PRO_PATH%\envs\%ENV_NAME%\Scripts"

Re: Distributing virtual environment to coworkers

Glasnoct — Thu, 30 Jun 2022 19:02:14 GMT

I spoke too soon it seems as this method doesn't totally work. If I try to run it twice in a row (to check that it find all the installed modules on the second run) it still installs them again. If I run the code and then at the end just attempt to import the installed modules it throws an error about being unable to find first one it comes across. No idea if that's an issue with trying to install modules without, like, restarting Arc or what.

Also had an issue with getting some of the dependencies installed for the modules. Pywin32 seems to need admin rights to install, which is a nonstarter if the script is being run by the end user as they won't be running it with admin rights. pip with '--user' is said to work but I don't know if it IS or not to be honest.

Re: Distributing virtual environment to coworkers

Anonymous User — Fri, 01 Jul 2022 15:12:30 GMT

That is a bummer! If you want to revisit the bat file, here it is in a complete version. The user/ admin thing is probably due to trying to write to C:/Program Files... with all the cloned environments that there might be, its hard to say where the packages are trying to get installed. That is one reason I went for creating an environment and setting it for my stuff, and for getting the packages that the user may have installed from their clone. Wish there was an easier way for this....

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: VARIABLES :
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
@echo off

SETLOCAL

SET "ENV_NAME=assessors-env"

SET "PRO_PATH=C:\Program Files\ArcGIS\Pro\bin\Python"
SET "REQ_PATH=\\path to the requirements.txt"
SET PATH=%PATH%;"C:\Program Files\ArcGIS\Pro\bin\Python\Scripts"
SET PATH=%PATH%;"%PRO_PATH%\envs\%ENV_NAME%\Scripts"

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: COMMANDS :
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

REM start by activating the arcgis pro conda env
call "C:\Program Files\ArcGIS\Pro\bin\Python\Scripts\activate.bat" & (

REM update the new environment with pip packages.
pip install -r "%REQ_PATH%\requirements.txt"
ECHO ^-^-^> %ENV_NAME% environment updated pip
)

exit

Re: Distributing virtual environment to coworkers

Anonymous User — Wed, 06 Jul 2022 17:48:03 GMT

I created an ArcGIS Idea for to cover stuff like this. I'd appreciate any thumbs up/ comments/ support for it so we can get some solid guidance.

create documentation for esri's conda