Multiprocessing in Stand-alone Script

853
5
08-25-2020 12:41 PM
MKF62
by
Occasional Contributor III

I recently switched from ArcMap to ArcGIS Pro and am having trouble getting multiprocessing to work in an old script. It worked at one point in ArcMap, but recent testing shows it no longer works with a Python 2.7 interpreter either, so it needs to be overhauled in a way I can't figure out. Literally the only reason I need to run this in a separate process is because calling a script tool (TableToExcel) in a standalone script while debugging means none of the breakpoints will be hit after I run that line in my IDE (Debugging with ArcPy breakpoints after Point to Line). Any insight on how to get this working a separate process would be appreciated...

import os, arcpy, multiprocessing

def tbl2excl(lyr, path):
    arcpy.TableToExcel_conversion(lyr, path, 'ALIAS')

arcpy.env.overwriteOutput = True
if __name__ == '__main__':
    #Get the data and do a couple things
    fileName = now.strftime("%Y-%m-%d")
    p = multiprocessing.Process(target=tbl2excl, args=(countSelection, os.path.join("E:\GIS_Testing\xxx", fileName + "_Counts.xls")))
    p.start()
    p.join()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

It just gets hung up after running p.join() and never stops. I assume it's erroring out in the new process somewhere, but I can't bring those error messages up to the parent script despite my efforts. If I don't try to run it in a separate process it doesn't error, so it's something specific to the multiprocessing, not using the TableToExcel tool itself.

Tags (1)
0 Kudos
5 Replies
DanPatterson
MVP Esteemed Contributor

If that is the actual path, then that is the problem... use raw encoding

p = "E:\GIS_Testing\xxx"
  File "<ipython-input-1-018658955c3a>", line 1
    p = "E:\GIS_Testing\xxx"
       ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 14-15: truncated \xXX escape

... sort of retired...
0 Kudos
MKF62
by
Occasional Contributor III

Sorry - it's not the actual path, I just didn't want to put the whole thing in.

0 Kudos
forestknutsen1
MVP Regular Contributor

But Dan still has a point. I always use raw encoding for paths.

r'c:\xxxx\xxxx\xxxx'
0 Kudos
DanPatterson
MVP Esteemed Contributor

Well, lets move on to the other missing bit of info...

countSelection

... sort of retired...
0 Kudos
JoshuaBixby
MVP Esteemed Contributor

For debugging, it might be best to start with subprocess — Python 3.8.5?   and graduate to multiprocessing once you know it works as a separate process.

0 Kudos