Accessing and cloning python environment with Pro SDK

[Anonymous User]

I'm trying to create a configuration/ add-in for a department and because Pro invalidates the cloned python environments after updates, I need a way to ensure that a cloned environment is created and the required python packages (one being our in-house package) are installed. Since some packages do not install through the Pro's conda manager, I need to install through pip.  I cannot ask each user to execute pip commands after each update of Pro. Nor do I really want to touch each computer after each update.

I haven't been able to find any examples of accessing the cloned environments, or if the python/ conda environment is exposed to the ArcGIS Pro SDK. A similar question without an answer was asked here.  I created a script that checks for the packages I need to install and installs them if they are not, but it needs the cloned environment first.

Simply, I need to do this:

User starts Pro after an update or fresh install.

Be able to:

1. Check if the python env is cloned and clone it if isn't.

  1.a. If it exists, check if python packages are installed and install if they are not.

2. Set active python environment to the clone that contains added packages.

3. Run python scripts from the addin buttons.

If this is not possible to do programmatically through the SDK, I'd welcome any options.  Thanks!

[Anonymous User]

I manage my non-ESRI conda envs through a yaml requirements file. That tends to simply things, because you can rebuild with a one liner. I have not tested this approach with conda envs in ArcGIS Pro, but would be curious if it works for you.

[Anonymous User]

Thanks, I discounted the yaml route because there wasn't a way (back then) that I could find to set the Pro project's environment to environments that were created outside of Pro's own conda manager. But since then, what's-new-with-python-in-arcgis-pro-2-8 demonstrates that it is possible so I'll have to reconsider it as an option and point a button to a bat file. 

The 6th step Hannes provided is what I needed:

1. Generate a list of additional packages installed in your current environment,
   with the conda command: 
   >>> conda env export > env.yaml
2. (Optional) If you have additional dependencies installed through pip,
   find those with pip freeze:
   >>> pip freeze > requirements.txt
3. Create a new environment by cloning arcgispro-py3, and activate it:
   >>> conda create --clone arcgispro-py3 --name my-env --pinned
   >>> activate my-env
4. Add back missing conda packages from your current environment:
   >>> conda env update -n my-env -f env.yaml
5. (Optional) Add back missing packages from pip:
   >>> pip install -r requirements.txt
6. (Optional) Make it the default for the ArcGIS Pro application and the
   "Python Command Prompt":
   >>> proswap my-env

[Anonymous User]

Thanks for the good link and good workflow. I might have to give this a second shot in Pro as well.

Worth noting that you can pip install from within the YAML file as well. This eliminates the needs for a requirements.txt and guarantees that the pip installed packages make it to the correct conda env (incase there is somehow more than one pip installed).

[Anonymous User]

After some loss of hair, I cobbled a bat file together that performs the steps of cloning the default environment, installs missing packages, and sets the pro's active python env to the new env.  I am no CMD genius so if this can be improved, please feel free to do so and post it.  I got a lot of 700 errors from jupyter-notebook at times while developing this.  Not sure what the exact fix was but it seemed to dissappear when I cloned the default env and updated it with cloned env.  Gotta quit while I'm ahead.

The called scripts (activate.bet & proswap.bat) are using full paths because it would stop if I tried to activate the new env to run commands.

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: VARIABLES                                                                    :
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
@echo off

SETLOCAL
SET "ENV_NAME=assessors-env"
SET "ARCGIS_DEFAULT=arcgispro-py3"
SET "ARCGIS_CLONE=arcgispro-py3-clone"

SET "PRO_PATH=C:\Program Files\ArcGIS\Pro\bin\Python"
SET "CLONED_PATH=%LOCALAPPDATA%\ESRI\conda\envs"
REM Set the default script path for this iteration. Maybe fixes jupyter notebook install errors?
SET PATH=%PATH%;"C:\Program Files\ArcGIS\Pro\bin\Python\Scripts"

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: COMMANDS                                                                     :
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

REM start by activating the arcgis pro conda env
call "C:\Program Files\ArcGIS\Pro\bin\Python\Scripts\activate.bat" & (
	REM export the cloned arcgispro-py3-clone environment that contains the additional packages installed outside of arcgis pro
		conda env export -n %ARCGIS_CLONE% > "%CLONED_PATH%\%ARCGIS_CLONE%\environment.yml"	
		ECHO ^-^-^> %ARCGIS_CLONE% conda environment exported to "%CLONED_PATH%\%ARCGIS_CLONE%\environment.yml"
		
	REM clone the deafult arcgispro-py3 environment to the new environment name
		REM CALL conda config --add channels conda-forge
		REM CALL conda config --add channels esri
		conda create --clone %ARCGIS_DEFAULT% --name %ENV_NAME% --pinned
		ECHO ^-^-^> %ENV_NAME% environment created
		
	REM update the new environment with packages that are not installed from the environment.yml created earlier.
	    REM conda env list
		REM conda env update -n %ENV_NAME% -f "%PRO_PATH%\envs\%ARCGIS_DEFAULT%\environment.yml"
		conda env update -n %ENV_NAME% -f "%CLONED_PATH%\%ARCGIS_CLONE%\environment.yml"
		ECHO ^-^-^> %ENV_NAME% environment updated
	
	REM set the active environment in Pro to the new environment
		CALL "%PRO_PATH%\Scripts\proswap.bat" %ENV_NAME%
		ECHO ^-^-^> Pro env set
	)
pause
exit

Now to get it to run from C#.

[Anonymous User]

This is the c# part placed in the button onclick event. - I'm still playing with how to capture the messages or just let them display in the console window because I dont think the SDK has an output window other than the geoprocessing one. 

await QueuedTask.Run(async () =>
            {
                try
                {
                    ProcessStartInfo processInfo = new ProcessStartInfo();
                    processInfo.FileName = @"\\gisdata\Transfer_Tech\ESRISoftware\Addins\python-env.bat";
                    processInfo.UseShellExecute = false;
                    processInfo.CreateNoWindow = false;
                    processInfo.UseShellExecute = false;
                    processInfo.RedirectStandardError = false;
                    processInfo.RedirectStandardOutput = false;

                    using (Process process = Process.Start(processInfo))
                    {
                        process.OutputDataReceived += (object sender, DataReceivedEventArgs e) =>
                            MessageBox.Show("output>>" + e.Data);
                        process.BeginOutputReadLine();

                        process.ErrorDataReceived += (object sender, DataReceivedEventArgs e) =>
                            MessageBox.Show("error>>" + e.Data);
                        process.BeginErrorReadLine();

                        process.WaitForExit();

                        //MessageBox.Show("ExitCode: {0}", process.ExitCode);

                        process.Close();
                    }

                    // Show a messagebox with the results
                    MessageBox.Show("Python environment updated successfully.");

                }
                catch (Exception exc)
                {
                    // Catch any exception found and display in a message box
                    if (exc.Message != "StandardOut has not been redirected or the process hasn't started yet.")
                    {
                        MessageBox.Show($"Exception caught while trying to validate the python env: {exc.Message}.");
                    }
                    
                    return;
                }
            });

[Anonymous User]

I created an idea for providing conda documentation and how can we deploy environments using Pro SDK or python along with our configs and add-ins in case anyone reaches this and wants to help out.

create documentation for esri's conda