How to DevOps Build Pipeline with arcpy python module?

[MaximilianGlas]

Hey guys,

We are actually developing Web Tools based on ArcGIS Pro 2.8 to publish in ArcGIS Enterprise 10.8.1.

We want to establish unit tests and want to run these tests in Azure DevOps Pipelines.

While in a local environment I have an ArcGIS Pro installed, of course this is not the case in an Azure DevOps environment. And simply trying to install arcpy module with python install failed.

What is the recommended scenario from Esri to establish this?

Kind regards,

Max

[Anonymous User]

Does the arcgis for python API work for what you are trying to do?

[MaximilianGlas]

Well, we already use the arcgis for python API additionally, but it does not cover all our use cases. That's why we also need the arcpy module.

[ArnaudRodier]

Hi @MaximilianGlas ,

We would like to use arcpy library to run some python script and register views via Azure DevOps Pipelines. Did you get any answer from ESRI or did you manage to make it working?

Thanks

Arnaud

[MaximilianGlas]

I got no answer so far and I don't have a real solution.

Possible ideas:

• Install an ArcGIS product which delivers arcpy (Pro or Server) on an machine which is used in the build process. But: no idea if this will work. And you are dependent on this installation, which could change with an auto update. Not really preferred.
• Try to isolate arcpy (and its dependencies) and include it in the version control system. No idea if this is possible with an acceptable effort. At least it will be a lot of data (my arcpy has 1,15 GB) in vcs. Even not sure if this is feasible because of the underlying ArcObjects core.
• Try to mock the arcpy calls. In my tests this works fine, but of course you will not be able to test all components with real conditions.

What are your thoughts and attempts so far?

[LondonWalker]

Has anyone found a solution to this?

[BethClarke]

We attempted the following solution to get unit tests that use arcpy running on Azure DevOps via a Windows machine:

• Install ArcGIS Pro on the machine
• License ArcGIS Pro using a Single Use license (this should license ArcGIS Pro for all users)
• Configure our .condarc file to set the env_dirs variable to the location of any new ArcGIS conda environments, as they are not located in the default conda environment location of C:\ProgramData\Miniconda\envs
• use the 'set CONDA_ENVS_PATH=<path_to_envs_dirs>' command to enable the machine to find all available Conda environments 
• Create an empty conda environment by opening proenv.bat (in C:\Program Files\ArcGIS\Pro\bin\Python\Scripts) and then executing 'conda create --name <env_name> python==3.9.18'. This will just have Python installed.
• Add the environment to ArcGIS Pro by opening ArcGIS Pro > Settings > Package Manager > Environment Manager > Add existing environment. The environment will appear as 'broken' after adding it to ArcGIS Pro. Click on 'Repair broken environment'.
• Switch to your new environment and activate it by executing '"C:\Program Files\ArcGIS\Pro\bin\Python\Scripts\proswap.bat"<conda_env_name>' in a CMD terminal

When we do this, we get the following error when attempting to execute unit tests:

"C:\Custom_ArcGIS_Envs\esri-test\lib\site-packages\arcgisscripting\__init__.py", line 131, in <module>

from ._arcgisscripting import *
ImportError: DLL load failed while importing _arcgisscripting: The specified module could not be found.

[LongDinh]

Hi @MaximilianGlas,

I noticed there wasn't a solution to your question. This is all possible in Azure DevOps services and will require significant setup/understanding of Azure Pipelines.

Firstly,  some requirements. You will need:

• A machine with:
  • An account with Administrative privileges,
  • ArcGIS Pro installation with a Single use license allocated,
  • A Self-hosted Windows agent installed to your DevOps organization or instance.

Before installing your Self-hosted Windows agent, you should clone the default CONDA environment of your ArcGIS Pro installation. It will allow us to install any additional packages (like pytest-cov & pytest-azurepipelines. We can call it arcgispro-azure-agent-env. We should also set this as an agent capability by adding it as a Windows Environment variable. System or locally is fine. We will call it ARC_AZ_ENV.

Run through the installation guide for Self-hosted Windows agent. Your Azure DevOps organization/instance should now have access to your machine via the Agent under an Agent Pool.

Assuming your tests are in the root path of your Azure Repo, we can setup a build pipeline which runs a job with your Agent. The job below will pip install pytest-cov and pytest-azurepipelines, run the pytests from the root and finally, publish the coverage results.

jobs:
- job: TestWebTool
  pool:
    name: MyAgentPool
    demands:
      - ARC_AZ_ENV
  steps:
    - checkout: self
    - script: |
        pip.exe install pytest-cov
        pip.exe install pytest-azurepipelines
      workingDirectory: "$(ARC_AZ_ENV)\Scripts"
    - script: |
        "$(ARC_AZ_ENV)\python.exe -m pytest --cov --cov-report xml:converage.xml
      workingDirectory: $(Build.SourcesDirectory)
    - task: PublishCodeCoverageResults@2
      inputs:
        summaryFileLocation:$(Build.SourcesDirectory)\coverage.xml

Azure Pipelines will automatically pickup your pytest results and publish them. Azure DevOps services have a number of formats and test suites which can be published. You can use the task: PublishTestResults@2. The tests and coverage results will appear in your build result like below.

Good luck!

[BethClarke]

We ended up with a similar solution to @LongDinh. 

We have managed to get the ArcGIS license working on an Azure DevOps CICD pipeline. We used the following steps:

1. Install ArCGIS Pro (version 3.2) system wide on the machine where the Azure DevOps build agents will be located
2. License ArcGIS Pro using a Single Use license (this should license ArcGIS Pro for all users)
3. Identify the location of the ArcGIS Pro conda environment on the machine 
4. From your local IT team, request creation of a service account with a username and password to be added to the machine where the pipelines run 
5. Log into the machine with the service account 
6. Install an Azure DevOps build agent, making sure to run it as a service using the username and password of the new service account. Also set SID type to SERVICE_SID_TYPE_UNRESTRICTED = Y during configuration of the agent
7. Ensure that the agent has log on permissions for your machine  by going to Start->Administrative Tools->Local Security Policy. Navigate to Local Policies->User Rights Assignment->Allow Log On Locally, and add the usernames of the accounts/groups which require access. After that, select Deny log on as a service and make sure that the username of the service account is not listed here. If this is not done you will get log on errors when attempting to start the agent.
8. Whilst still logged in to your machine with the service account credentials, open ArcGIS Pro and log in to retrieve the license 
9. In your YAML file for configuring the Azure DevOps pipeline, ensure that the path to the ArcGIS Pro condabin directory is added to your PATH as an early pipeline step. 
10. Execute your Azure DevOps pipeline. Your tests should run and be able to pick up the license, allowing ArcPy to be used in your tests.