Distributing virtual environment to coworkers

[Glasnoct]

Our current toolbox of scripts requires the installation of some modules that do not come with ArcPro out of the box and the current solution has been to have coworkers fire up the Python command prompt and do a "pip install" for the missing modules but that is proving to be too advanced for them and thus I'm looking for a simpler solution.

What's the easiest way I can keep everyone up to date with any dependencies we need? Can I just do a copy-paste of the virtual environment folder? Could I clone the environment to a shared directory and have them point their project environment to that? That would be useful as I can update it as necessary and it will "populate" to all the other users.

[Anonymous User]

That is timely... I am working the same process for a department. I'll share what I did, but I am also very interested in seeing what others have done as well. I haven't really found a solid solution of combining/ deploying custom python environments along with add-ins, which seems they would go hand in hand. If you are making an add-in to run a custom script... most likely you'll have a cloned environment and packages that go along with it right? It would be awesome if esri could provide a means to create/ manage environments through the SDK.

I dread that when Pro updates, it will constantly invalidate the new environment. With this in mind, I need a way that the user can recreate the environment each time, without me having to touch each machine after each update. I also need to be able to capture packages that they might have imported for other scripts, so they don't have to constantly switch environments or have to re-install each time.

I came up with a bat file script that I run from an add-in button (it can be ran as a bat file from the desktop as well) to clone the pro default env, create a yml from a cloned env, install the missing packages, copy local in-house packages, and sets the active environment in Pro to the new env.  I've had some issues with it- jupyer-notebook seems to be a pita and fails to install sometimes, which causes the transaction to rollback.  numpy sometimes complains that it wasn't installed correctly on 1/4 of the pc's I've ran this on.

I'm at the stage of testing deployment and adjusting commands to see if I can get it working 100%, so the -freeze-installed --no-update-deps is trial to hopefully fix this.  I'm not a conda pro by any means, so if there is a better command I should be using, please feel free to let me know. I tried to comment the code to explain each step and when I have more time, I'd like to incorporate if statements for using the clone-1 environment if it exists but for now, this gets me 98% of the way there very quickly. You can grab the yml and save it where it can be accessed and install from there too by reference the path in the conda env update line.

I saved a requirements.txt that has a list of additional packages I need, (grabbed from the end of the yml) and do a second pip for these to make sure that they get installed.

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: VARIABLES :
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
@echo off

SETLOCAL

REM set environment names
SET "ENV_NAME=assessors-env"
SET "ARCGIS_DEFAULT=arcgispro-py3"
SET "ARCGIS_CLONE=arcgispro-py3-clone"

REM Paths set to variables
SET "PRO_PATH=C:\Program Files\ArcGIS\Pro\bin\Python"
SET "CLONED_PATH=%LOCALAPPDATA%\ESRI\conda\envs"
SET "REQ_PATH=\\gisdata\Assessor\ASR_ArcGIS Pro"

REM Set the default scripts dir path for this iteration for pip installing
SET PATH=%PATH%;"C:\Program Files\ArcGIS\Pro\bin\Python\Scripts"
REM Set the new env scripts dir path for this iteration for pip installing
SET PATH=%PATH%;"%PRO_PATH%\envs\%ENV_NAME%\Scripts"

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: COMMANDS :
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

REM start by activating the arcgis pro conda env. If successful, continue process
call "C:\Program Files\ArcGIS\Pro\bin\Python\Scripts\activate.bat" & (
REM export the cloned arcgispro-py3-clone environment that contains the additional packages installed outside of arcgis pro
conda env export -n %ARCGIS_CLONE% > "%CLONED_PATH%\%ARCGIS_CLONE%\environment.yml"
ECHO ^-^-^> %ARCGIS_CLONE% conda environment exported to "%CLONED_PATH%\%ARCGIS_CLONE%\environment.yml"

REM clone the deafult arcgispro-py3 environment to the new environment name. If needed, you can add specific package channels
REM CALL conda config --add channels conda-forge
REM CALL conda config --add channels esri
conda create --clone %ARCGIS_DEFAULT% --name %ENV_NAME% --pinned
ECHO ^-^-^> %ENV_NAME% environment created

REM install packages that were not installed from the default clone, but were in the arcgispro-py3-clone environment.
REM conda env list
conda env update -n %ENV_NAME% -f "%CLONED_PATH%\%ARCGIS_CLONE%\environment.yml --freeze-installed --no-update-deps"
ECHO ^-^-^> %ENV_NAME% environment updated

REM ensure that the new env is updated with the pip installed packages at the end of the environment.yml file.
pip install -r "%REQ_PATH%\Support Scripts\requirements.txt"
ECHO ^-^-^> %ENV_NAME% environment updated pip

REM copy local packages to the env
robocopy "%REQ_PATH%\cntytools" "%PRO_PATH%\envs\%ENV_NAME%\Lib\site-packages\cntytools" /E

REM set the active environment in Pro to the new environment
CALL "%PRO_PATH%\Scripts\proswap.bat" %ENV_NAME%
ECHO ^-^-^> Pro env set
exit
)

exit

[Anonymous User]

This is the daml for the add-in button. If you look at the Ribbons control solution in Pro SDK github repo, you'll see how to add ribbons and buttons to an add in.

			<tabs>
				<tab id="RibbonControls_Tab1" caption="Admin Tools">
					<group refID="RibbonControls_PythonEnv" />
				</tab>
			</tabs>

			<groups>
				<!-- comment this out if you have no controls on the Addin tab to avoid
              an empty group-->
				
				<group id="RibbonControls_PythonEnv" caption="Python Env">
					<!-- host Assessor GIS Maintenance controls within groups -->
					<button refID="RibbonControls_PythonEnv_Btn" size="large"/>
					<!--Use subgroups to better control ribbon resizing-->
					<!--<subgroup refID="RibbonControls_Subgroup2" />-->
				</group>
			</groups>
			<controls>
				<!-- add your controls here -->
				<!-- host python environment buttons -->
				<button id="RibbonControls_PythonEnv_Btn" caption="Python Environment" className="PythonEnv_Btn" loadOnClick="true" smallImage="Images\Python-PNG-Clipart.png" largeImage="Images\Python-PNG-Clipart.png">
					<tooltip heading="Establish Assessors-env python env">
						Creates the assessors-env python environment and installs the required packages.<disabledText />
					</tooltip>
				</button>
			</controls>

This is the button logic for the add-in to run the bat file. The bat file is added as content, always copied as the pro snippets documentation for files suggest.

internal class PythonEnv_Btn : Button
    {
        protected async override void OnClick()
        {
            var batFile = Path.Combine(Path.GetDirectoryName(Uri.UnescapeDataString(new Uri(Assembly.GetExecutingAssembly().CodeBase).LocalPath)), @"Utils\python-env.bat");

            await QueuedTask.Run(async () =>
            {
                try
                {
                    ProcessStartInfo processInfo = new ProcessStartInfo
                    {
                        FileName = batFile,
                        UseShellExecute = false,
                        CreateNoWindow = false
                    };
                    processInfo.UseShellExecute = false;
                    processInfo.RedirectStandardError = false;
                    processInfo.RedirectStandardOutput = false;

                    using (Process process = Process.Start(processInfo))
                    {
                        process.WaitForExit();
                        process.Close();
                    }

                    // Show a messagebox with the results
                    MessageBox.Show(@"Check if assessors-env was created at C:\Program Files\ArcGIS\Pro\bin\Python\envs.");

                }
                catch (Exception exc)
                {
                    // Catch any exception found and display in a message box
                    MessageBox.Show($"Exception caught while trying to validate the python env: {exc.Message}.");

                    return;
                }
            });
        }
    }

It opens a cmd and runs the bat file. When you see (env name) C:> stuff/stuff/stuff> , you can close the window.  It doesn't take any more instructions from the bat file after it activates the new environment and I haven't found a way to close it yet.

[Anonymous User]

Ok, last one but I just thought of it after submitting the last post. You could temporary set the paths to a python package library location at the beginning of your scripts to point to the additional packages you need.

os.environ["PATH"] = 'path to your packages'

[Glasnoct]

Lordy that's a lot of info! My use case doesn't involve any add-ins since that's a whole can of worms I doubt I'll ever open. It looks like the .bat file you have there also involves cloning the default environment. My coworkers already have cloned environments so I'm thinking the easiest way forward would just be to have them run a batch file that tries to install all the required modules. This way I can update the batch file when I have new requirements and all I need is them to run it (or even better, maybe it could run as part of the tool script? Could I do that?)

What's the barebones batch file I can distribute to them that'll just do a pip install of a module list via their currently active environment?

[Anonymous User]

True, It’s a lot because it has to do a lot so I do t have to do a lot. 🙂

Ill be away for a while without internet but barebones is these two lines- replace the %variable% with your path to the requirements.txt file.

Files\ArcGIS\Pro\bin\Python\Scripts\activate.bat" & (
REM ensure that the new env is updated with the pip installed packages at the end of the environment.yml file.
pip install -r "%REQ_PATH%\Support Scripts\requirements.txt"
ECHO ^-^-^> %ENV_NAME% environment updated pip)

[Glasnoct]

Did a bit of your code get chopped off there? %REQ_PATH% isn't declared prior to usage (or doesn't need to? I don't really .bat) and it starts off with "Files".

edit: I gotcha; I just realized it's snipped from the original batch file. I'll piece it together.

edit2: Nope, played with it a bit but can't get it to work. activate.bat complains about "pip" and if I try "python pip" instead it complains that python can't be found. Trying it with proenv.bat activates fine but the command doesn't pass through and nothing on Google is answering how to pass the pip command after calling proenv.bat

[Anonymous User]

I was posting through my phone and had to keep it simple.  Geonet isn't very mobile friendly and I am glad that it made enough sense for you to at least try it, but I don't think I copied enough for it to work.

I came into that issue too with pip, activate.bat and proenv.bat as well. To get around it (and what I didn't grab to paste in the last post), I set a temp environment variable pointing to where the scripts directory is for pip.exe:

SET PATH=%PATH%;"C:\Program Files\ArcGIS\Pro\bin\Python\Scripts"

SET PATH=%PATH%;"%PRO_PATH%\envs\%ENV_NAME%\Scripts"

[Glasnoct]

For future reference this is what I arrived at finally. stick it in a .py and import it as a module into your script. Edit the required variable to include the names of any modules that will need to be pip installed if not already. No .bat files required and installs to the user's current active conda environment.

import arcpy
import sys
import subprocess
import pkg_resources


required = {'excel2img', 'pypdf2', 'ahk', 'ahk-binary', 'pysimplegui'}


def check_required_modules(mod_set: set)-> None:
    """
    Compares input set against current venv installed modules and installs them if missing
    e.g. mod_list = {'excel2img', 'pypdf2', 'ahk', 'ahk-binary', 'pysimplegui'}
    :param mod_set: string set of module names to check
    :return:
    """

    installed = {pkg.key for pkg in pkg_resources.working_set}
    missing = mod_set - installed

    if missing:
        python = r"c:\Progra~1\ArcGIS\Pro\bin\Python\scripts\propy.bat"
        for required_module in missing:
            subprocess.check_output([python, '-m', 'pip', 'install', required_module], shell=True)
            msg = f"{required_module} was missing and was installed"
            print(msg)
            arcpy.AddMessage(msg)
    else:
        msg = 'all required modules are installed! :)'
        print(msg)
        arcpy.AddMessage(msg)
    sys.exit()


check_required_modules(required)

[Anonymous User]

That is a good solution, glad you were able to get something figured out!