Installation of additional Python modules into FME Workbench

[ThomasBeckerDK]

I need to install scipy into the Data Interoperability Workbench and tried the approach one would take with FME

fme.exe python -m pip install scipy==1.9.3

However, the whole thing blows up with the last error message ModuleNotFoundError: No module named '_socket'.

How can I install scipy? @Bruce_RHarold, any suggestion? 😉

[BruceHarold · Accepted solution]

Hello Thomas

If you clone your ArcGIS Pro 3.4 python environment then extend it with scipy, and set the runtime environment to the clone, and in Workbench translation options set your preferred Python interpreter to Esri ArcGIS Python 3.11, scipy will be available.

[ThomasBeckerDK]

Hi @BruceHarold,

thanks for your reply and initially it also worked somehow, but... it does not anymore.

Following your advice on how to install additional Python modules to use them in the Data Interoperability Workbench I installed SQLAlchemy and as a dependency also psycopg2. 

Here the screenshots from ArcGIS Pro where the active environment and the packages in question are shown with version numbers.

The Python Compatibility is set to Esri ArcGIS Python 3.11,
Esri ArcGIS Compatibility is set to Esri ArcGIS Server/ArcGIS Pro,
the necessary packages confirmed to be installed in the clone of ArcGIS Pro Python environment,
and the clone is set as the active environment.

Yet, I am getting 

The message about the Python interpreter used drives me mad... it does not point at the clone that is set as the active environment, but to the original Esri Python environment.

Using Python interpreter from `C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python311.dll' with PYTHONHOME `C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3'

I just checked my environment variables in Windows and I do not have a PYTHONHOME variable set...

I went into the FME options of the workbench and set there the path for the preferred Python interpreter to C:\Users\tsbr\AppData\Local\ESRI\conda\envs\arcgispro-extendable\python311.dll, and the PYTHONHOME variable to C:\Users\tsbr\AppData\Local\ESRI\conda\envs\arcgispro-extendable.

Still the message from the translation log is pointing to the arcgispro-py3 environment.

[BruceHarold]

Hi Thomas, I also see the default environment path logged, but in my case the cloned environment is available, let us look into this.  Back ASAP.

[BruceHarold]

Thomas, while we're looking at this, to get going you can explicitly set the interpreter:

https://community.safe.com/general-10/how-do-i-direct-fme-to-use-the-active-cloned-python-environmen...

[ThomasBeckerDK · Accepted solution]

Morning Bruce,

thanks for the link! So I had been half the way to success with my last posting. 😉
The important part when setting the preferred Python interpreter under 'Translation' in the FME Options is that your workspace has to have the Python Compatibility set to a non-Esri ArcGIS Python 3.x option as FME will load ArcGIS' default Conda environment and ignore the custom interpreter settings from FME Options.

Adjusting the Python Compatibility for the workspace to Python 3.11+ did the trick.
Subsequently I changed the preferred Python interpreter in the FME Options also to Python 3.11+, thereby removing the direct reference to the specific clone, and removed the setting for Python Home entirely.

A test run of the workspace did not show any issues with missing packages. My essential take-away from this is that a Python Compatibility setting to ESRI ArcGIS Python x.x is a fallback to the default Python environment in ArcGIS Pro. Using the active Python environment from ArcGIS Pro requires a setting of the Python compatibility to Python 3.11+ (being currently on ArcGIS Pro 3.4). 

[BruceHarold]

Thanks for the clarification Thomas, we'll look into making this behavior easier to use.