Publishing Geoprocessing Service based on Python 3 to ArcGIS Enterprise on Linux

[MatthewLevey]

Using ArcGIS Pro 2.8 I successfully published a geoprocessing script written in Python 3 as a web tool to our Federated Portal running ArcGIS Enterprise 10.9 for Linux. I had cloned the arcgispro-py3 environment and added the Python packages needed to run the script.

Attempting to run a job with that gp service fails because of the missing Python libraries. The documentation says that third-party Python 3 packages can't be installed on Linux, but that can't be entirely true because that would leave too many users unable to utilize their expensive software and hardware. Libraries I need are no longer available for Python 2.7.

My questions:

1. Does Portal use the arcgispro-py3 environment running under Wine? If so what is the correct process to clone that environment? My attempts have failed because of jupyter-notebook scripts failing.
2. Is it safe to add packages to the arcgispro-py3 environment on Server? If I try to add packages to the default environment Conda wants to do a lot up updating for packages not in the '.pinned' file. 

I am really stuck here. Thanks in advance or any help or advice!

[JonEmch]

Hello Matthew, thank you for reaching out.

   Unfortunately, the third party Python libraries limitation very much exists on Linux environments. As mentioned, it is documented here:

"For geoprocessing services published from ArcGIS Pro or the arcpy module based on your ArcGIS Pro conda environment, third-party Python packages cannot be deployed on your Linux server." - Deploying custom Python packages for ArcGIS Server

However, there may be pathing forward for you here: Python 3 runtime for ArcGIS Server on Linux

This document goes into some detail on how to use a Python 3 runtime for command line ArcPy scripts. Select tools can be executed with a Python 3.x runtime that is installed independently of ArcGIS Server.

[MatthewLevey]

Hi Jon,

Thank you for your reply.

I really don't understand the logic of not supporting Python 3 geoprocessing on Server for Linux while at the same time pushing us toward using ArcGIS Pro, which uses Python 3. ArcMap is not being updated and Python 2.7 development was discontinued over a year ago and the libraries we need have moved on. This leaves those of us using Linux without a solution and from the number of views for this post, there are many of us.

I am aware of using Python 3 from the command line. That is a solution for a different problem. If I am going to have to ask my users to utilize the command line I will just create a Flask app and bypass Server altogether. Perhaps we don't even need ArcGIS Enterprise at all?

This situation has left me very disgruntled. Despite the phrase in your signature I can't keep on keeping on, I am dead in the water!

[MatthewLevey · Accepted solution]

I have found a solution and one that is so simple I wish I had thought of it earlier. The workaround for not being able to install third-party Python libraries on Linux is to import them from another folder, in my case another conda environment that I created. Here are the steps that I used:

1. On my local computer running ArcGIS Pro I cloned the arcgispro-py3 environment and installed the libraries I needed.
2. On the Linux server running ArcGIS Server I used conda to create an environment and using conda installed arcgis-server-py3 and the libraries I needed. 
3. In my main script I imported sys and added two lines. Since apparently the ArcGISPro Runtime still runs on Wine, "Z:" is the drive for the server path:
   

   sys.path.insert(-1, "\\UNC path to server\path\to\server\environment\site-packages")
   sys.path.insert(-1, "Z:\path\to\server\environment\site-packages")

4. These lines were added after other library imports and the paths appended to the end of the list (hence the -1) so that libraries in the active environment are loaded first.
5. I ran the geoprocessing script in ArcGIS Pro then shared the result as a Web Tool.
6. Finally I pointed my web app to the REST endpoint for the gp service.

I am still trying to get the multiprocessing to work (tricky since it runs under Windows emulation) but other than that, it works. Hopefully this will help someone else who uses ArcGIS Enterprise on Linux and needs to use Python libraries not included in the default install.

[NilsBabel1]

Hi Matt, I'm having the exact same issue.  Could you provide a little more detail to your steps 1 and 2 above?  How exactly do you clone an environment and how do you install that environment onto the Linux server?

Thanks so much.  And I totally agree, this seems like a complete oversight.  How can we run Python3 and ArcGIS Server on Linux if we can't add python modules?

Nils

[MatthewLevey]

Hi Nils,

My post is a bit dated and we have since moved on to an open-source solution. Both steps 1 and 2 are both fully explained in the Esri online documentation for ArcGIS Pro and ArcGIS Enterprise for Linux.

To summarize though: on your desktop you will clone the default arcgispro-py3 environment using that option under the Python menu in ArcGIS Pro. Install your needed Python libraries to that newly cloned environment. Then on your Linux machine, you will create a new Python virtual environment using whatever method you wish. I used Conda. Add the Esri channel (similar to how one adds conda-forge - see the docs) and then install arcgis-server-py3. Perhaps obvious, but you must have ArcGIS Server or Enterprise installed and licensed on that Linux machine. Then add the Python libraries you need to that environment and point that environment in your scripts as above.

I hope this helps but again, the Esri documentation should get you through those two steps.

I wish you good success,

Matt