topic Re: Multiprocessing in Stand-alone Script in Python Questions

Multiprocessing in Stand-alone Script

MKF62 — Sat, 11 Dec 2021 14:13:46 GMT

I recently switched from ArcMap to ArcGIS Pro and am having trouble getting multiprocessing to work in an old script. It worked at one point in ArcMap, but recent testing shows it no longer works with a Python 2.7 interpreter either, so it needs to be overhauled in a way I can't figure out. Literally the only reason I need to run this in a separate process is because calling a script tool (TableToExcel) in a standalone script while debugging means none of the breakpoints will be hit after I run that line in my IDE (Debugging with ArcPy breakpoints after Point to Line). Any insight on how to get this working a separate process would be appreciated...

import os, arcpy, multiprocessing

def tbl2excl(lyr, path):
    arcpy.TableToExcel_conversion(lyr, path, 'ALIAS')

arcpy.env.overwriteOutput = True
if __name__ == '__main__':
    #Get the data and do a couple things
    fileName = now.strftime("%Y-%m-%d")
    p = multiprocessing.Process(target=tbl2excl, args=(countSelection, os.path.join("E:\GIS_Testing\xxx", fileName + "_Counts.xls")))
    p.start()
    p.join()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

It just gets hung up after running p.join() and never stops. I assume it's erroring out in the new process somewhere, but I can't bring those error messages up to the parent script despite my efforts. If I don't try to run it in a separate process it doesn't error, so it's something specific to the multiprocessing, not using the TableToExcel tool itself.

Re: Multiprocessing in Stand-alone Script

DanPatterson — Sat, 11 Dec 2021 14:13:49 GMT

If that is the actual path, then that is the problem... use raw encoding

p = "E:\GIS_Testing\xxx"
  File "<ipython-input-1-018658955c3a>", line 1
    p = "E:\GIS_Testing\xxx"
       ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 14-15: truncated \xXX escape‍‍‍‍‍

Re: Multiprocessing in Stand-alone Script

MKF62 — Tue, 25 Aug 2020 19:54:00 GMT

Sorry - it's not the actual path, I just didn't want to put the whole thing in.

Re: Multiprocessing in Stand-alone Script

forestknutsen1 — Tue, 25 Aug 2020 20:01:54 GMT

But Dan still has a point. I always use raw encoding for paths.

r'c:\xxxx\xxxx\xxxx'‍

Re: Multiprocessing in Stand-alone Script

DanPatterson — Tue, 25 Aug 2020 23:11:11 GMT

Well, lets move on to the other missing bit of info...

countSelection

Re: Multiprocessing in Stand-alone Script

JoshuaBixby — Tue, 25 Aug 2020 23:18:53 GMT

For debugging, it might be best to start with subprocess — Python 3.8.5? and graduate to multiprocessing once you know it works as a separate process.