Geoprocessing from external python IDE

Question

I have a substantial amount of code that I need to run through ArcGIS Pro. Due to the length, I decided to switch to an IDE that has a good debugger and I installed PyCharm 2021.1.1. I have not had any success running any geoprocessing commands as I always get "RuntimeError: Object: Error in executing tool" and/or "python-BaseException" errors. I wrote a simple code to create a buffer around a polygon feature class and it has the same errors (I presume once I get this code fixed, my longer code will also work). Does anyone know what I can change to get this to work?

I have ArcGIS Pro 2.8 and the commands run just fine there. I have the same python interpreter set in both ArcGIS and PyCharm (it is a clone of the default with a couple packages added). I leave ArcGIS Pro open while I run the code so it will automatically load the license (which I have checked and it does that properly). I know my shapefile reader is also working (at least I can pull values off my shapefiles just fine). I also get an error code that the mkl-service package is not importing correctly; I don't know if that is the problem but I haven't been able to figure it out. I've uninstalled and reinstalled the mkl-service package and restarted both PyCharm and ArcGIS Pro. I'm pretty new to Python and not confident on the import statements; I know I don't need all of them for this code but I think I need them all for pieces of my bigger code so they are all included here.

My code (Temp.py):

from datetime import date
from datetime import time
from datetime import datetime
import array as arr
import arcpy
from arcpy import env
from arcpy import analysis
arcpy.CheckOutExtension("Spatial")
import shapefile
import numpy
import csv
import sys

ThisFolder = "D:\LHD_GIS\ResearchAttempt5Python\"
in_raster = 'D:\LHD_GIS\ResearchAttempt5Python\ImagesForMyCode\Extract Bands_Gr5_12E.tif' ### will change
in_class_data = shapefile.Reader(ThisFolder + "LHD_Poly_all_TileName") ### may change
in_no_dams = shapefile.Reader(ThisFolder + "NotDams")

arcpy.analysis.Buffer(in_no_dams, 'D:\LHD_GIS\ResearchAttempt4\ResearchAttempt4.gdb\NotDams_Buffer2', "10 Feet", "FULL", "ROUND", "NONE", None, "PLANAR")

#This is the command I will need to use eventually:
#arcpy.ia.ExportTrainingDataForDeepLearning('D:\LHD_GIS\ResearchAttempt5Python\ImagesForMyCode\Extract Bands_group308.tif', "D:\LHD_GIS\ResearchAttempt4\TestExports\NonDam_May20_3", in_class_data, "TIFF", 1024, 1024, 512, 512, "ONLY_TILES_WITH_FEATURES", "PASCAL_VOC_rectangles", 0, "Classvalue", 0, None, 0, "MAP_SPACE", "PROCESS_AS_MOSAICKED_IMAGE", "NO_BLACKEN", "FIXED_SIZE", None)

print("We did it")

Message when I run the code:

Message after I run the debugger:

that top line of code that gets cut off in both images says: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service

Brian_Wilson · Answer

"Activating" changes the interpreter, basically when you clone an environment, it copies everything from the source into a new directory tucked away somewhere in an "envs" folder. Then activating points everything at that folder so it can have both Python libraries and programs that are specific only to that environment. When you start a python interactive session you should be able to just type "import mkl" and get no error or else get ModuleNotFound depending on whether the module is installed in the currently active environment.

I keep an osgeo environment around that has open source gis tools in it, that keeps it separate from the ones that ship with Pro.  I usually set up different environments for large projects, for example I have a 'covid' one for doing all the things required to collect and process COVID data for dashboards.

Caitlintr · Answer

Thanks Davin for the reminder! I used to have it like that, but I can't read values off the attribute table without the shapefile.reader. I'll create a new variable for when I want to see the attribute table and leave the in_no_dams variable as just the file path.

Caitlintr · Answer

Thank you Brian! I didn't learn the difference between an environment and an interpreter till yesterday and I'm still not sure I'm getting it right. I've also used that link that you've shared and it looks like it sets both the environment and the interpreter at the same time. I know I at least have one of them, if not both the environment and interpreter, set to my cloned ESRI environment and the python interpreter in that file. I haven't switched to it from the command line - I do it all within settings in PyCharm and within ArcGIS Pro (which I agree, that one is clunky). I've also tried uninstalling and reinstalling mkl-service, but it always gives me the same message. I will try activating mkl_test (which I have installed but I haven't touched).