Geoprocessing from external python IDE

1355
5
05-20-2021 02:16 PM
Caitlintr
New Contributor II

I have a substantial amount of code that I need to run through ArcGIS Pro. Due to the length, I decided to switch to an IDE that has a good debugger and I installed PyCharm 2021.1.1. I have not had any success running any geoprocessing commands as I always get "RuntimeError: Object: Error in executing tool" and/or "python-BaseException" errors. I wrote a simple code to create a buffer around a polygon feature class and it has the same errors (I presume once I get this code fixed, my longer code will also work). Does anyone know what I can change to get this to work?

I have ArcGIS Pro 2.8 and the commands run just fine there. I have the same python interpreter set in both ArcGIS and PyCharm (it is a clone of the default with a couple packages added). I leave ArcGIS Pro open while I run the code so it will automatically load the license (which I have checked and it does that properly). I know my shapefile reader is also working (at least I can pull values off my shapefiles just fine). I also get an error code that the mkl-service package is not importing correctly; I don't know if that is the problem but I haven't been able to figure it out. I've uninstalled and reinstalled the mkl-service package and restarted both PyCharm and ArcGIS Pro. I'm pretty new to Python and not confident on the import statements; I know I don't need all of them for this code but I think I need them all for pieces of my bigger code so they are all included here.

My code (Temp.py):

from datetime import date
from datetime import time
from datetime import datetime
import array as arr
import arcpy
from arcpy import env
from arcpy import analysis
arcpy.CheckOutExtension("Spatial")
import shapefile
import numpy
import csv
import sys

ThisFolder = "D:\\LHD_GIS\\ResearchAttempt5Python\\"
in_raster = 'D:\\LHD_GIS\\ResearchAttempt5Python\\ImagesForMyCode\\Extract Bands_Gr5_12E.tif' ### will change
in_class_data = shapefile.Reader(ThisFolder + "LHD_Poly_all_TileName") ### may change
in_no_dams = shapefile.Reader(ThisFolder + "NotDams")

arcpy.analysis.Buffer(in_no_dams, 'D:\\LHD_GIS\\ResearchAttempt4\\ResearchAttempt4.gdb\\NotDams_Buffer2', "10 Feet", "FULL", "ROUND", "NONE", None, "PLANAR")

#This is the command I will need to use eventually:
#arcpy.ia.ExportTrainingDataForDeepLearning('D:\\LHD_GIS\\ResearchAttempt5Python\\ImagesForMyCode\\Extract Bands_group308.tif', "D:\\LHD_GIS\\ResearchAttempt4\\TestExports\\NonDam_May20_3", in_class_data, "TIFF", 1024, 1024, 512, 512, "ONLY_TILES_WITH_FEATURES", "PASCAL_VOC_rectangles", 0, "Classvalue", 0, None, 0, "MAP_SPACE", "PROCESS_AS_MOSAICKED_IMAGE", "NO_BLACKEN", "FIXED_SIZE", None)

print
("We did it")

Message when I run the code:

Caitlintr_0-1621544811138.png

Message after I run the debugger:

Caitlintr_1-1621544867339.png

that top line of code that gets cut off in both images says: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service

0 Kudos
5 Replies
Brian_Wilson
Occasional Contributor III

Do you understand about conda and environments?

I use MS Visual Studio Code not PyCharm because it's just better. 🙂  But it looks like PyCharm includes support for conda, see https://www.jetbrains.com/help/pycharm/conda-support-creating-conda-virtual-environment.html

I keep separate environments for different projects, in a command shell when I do "conda env list" I see 

base                  *  C:\Program Files\ArcGIS\Pro\bin\Python
arcgispro-py3            C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3
accela                   C:\Users\bwilson\AppData\Local\ESRI\conda\envs\accela
arcgispro27-vscode       C:\Users\bwilson\AppData\Local\ESRI\conda\envs\arcgispro27-vscode
arctic                   C:\Users\bwilson\AppData\Local\ESRI\conda\envs\arctic

I would need to activate the one called "arcgispro27-vscode" because it has some things added for me to run in VSCode but the one installed by Esri would work too, and you'd probably want to specify that in PyCharm.

From the command line, I'd do this to use the standard Esri provided environment:

$ conda activate arcgispro-py3
(arcgispro-py3)
bwilson@ccis4501 MINGW64 ~
$ which python
/c/Program Files/ArcGIS/Pro/bin/Python/envs/arcgispro-py3/python
(arcgispro-py3)
bwilson@ccis4501 MINGW64 ~

 

This is bash shell, sorry, I'm a linux geek but what I am trying to show you is that when you activate a conda environment you will see that it actually also changes which version of python I use and also all the packages. So, before doing "conda activate arcgispro-py3" I can run "python" and from the >>> prompt if I do "import arcpy" it fails. AFTER activating, it succeeds.

The mkl-service package is still missing though. (I tried to do an "import mkl" to test it.)

I _could_ install it right away but I don't want to load random packages into the standard Esri environment (and it would require admin permissions anyway) so first I clone the environment, and then install the package in the clone. 

The commands would look like this, first I clone the Esri environment, then I activate the clone, then I add the mkl-service package, then I test it. I left out several hundred lines of what packages are installed. Lots.

conda create --name=mkl_test --clone=arcgispro-py3
conda activate mkl_test
conda install mkl-service
python
Python 3.7.9 [MSC v.1922 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mkl
>>> mkl.__version__
'2.3.0'
>>>

Now I have a "mkl_test" environment and I could open up my IDE (PyCharm in your case) and tell it that's the environment (python plus some other scripts and command plus the python modules). In theory you can manage your conda environments inside ArcGIS Pro but personally I found it too clunky slow and unpredictable. Maybe they fixed it, I have not tried it in 2.7. In Pro you will find in the Projects menu it's under "Python" in the left blue bar. Then see the "Manage Environments" button. You can clone and activate and install right there.

Does any of this make any sense? Let me know.

Caitlintr
New Contributor II

Thank you Brian! I didn't learn the difference between an environment and an interpreter till yesterday and I'm still not sure I'm getting it right. I've also used that link that you've shared and it looks like it sets both the environment and the interpreter at the same time. I know I at least have one of them, if not both the environment and interpreter, set to my cloned ESRI environment and the python interpreter in that file. I haven't switched to it from the command line - I do it all within settings in PyCharm and within ArcGIS Pro (which I agree, that one is clunky). I've also tried uninstalling and reinstalling mkl-service, but it always gives me the same message. I will try activating mkl_test (which I have installed but I haven't touched).

0 Kudos
Brian_Wilson
Occasional Contributor III

"Activating" changes the interpreter, basically when you clone an environment, it copies everything from the source into a new directory tucked away somewhere in an "envs" folder. Then activating points everything at that folder so it can have both Python libraries and programs that are specific only to that environment. When you start a python interactive session you should be able to just type "import mkl" and get no error or else get ModuleNotFound depending on whether the module is installed in the currently active environment. 

I keep an osgeo environment around that has open source gis tools in it, that keeps it separate from the ones that ship with Pro.  I usually set up different environments for large projects, for example I have a 'covid' one for doing all the things required to collect and process COVID data for dashboards.

0 Kudos
DavinWalker2
Esri Contributor

Hi,

There is no reason to use the shapefile.reader from the shapefile module.

Inputs to geoprocessing tools take the file path. This should work.

 

in_no_dams = ("D:\\LHD_GIS\\ResearchAttempt5Python\\NOT_DAMS.shp")

 

 

0 Kudos
Caitlintr
New Contributor II

Thanks Davin for the reminder! I used to have it like that, but I can't read values off the attribute table without the shapefile.reader. I'll create a new variable for when I want to see the attribute table and leave the in_no_dams variable as just the file path.

0 Kudos