Multiprocessing help

MKF62 — Sun, 12 Dec 2021 06:37:52 GMT

I'm trying to start a separate process to run the ExcelToTable tool since running that tool in the same process as the rest of the script breaks all breakpoints that come after that line (supposedly this bug has been known for years but apparently no one wants to fix it) and I still need to be able to use breakpoints so... how can I get this process to run? Since I just need to run this tool a single time and wait for it to finish, I'm using the Process method instead of the Pool method. When I run my script, I get an error about being unable to pickle a geoprocessing result object.

class DomainData(object):
    '''This is the base class for all domain data'''
    def __init__(self, folderPath, gdbPath, dbCon):
        #properties

    def createDomain(self, wsRows, tblName):
        #Make Excel file
        #Save the file
        wb.save(self.wkbkName)
        savedTable = os.path.join(self.gdb, tblName)
        #Convert the excel file to an ArcMap table.
        p = multiprocessing.Process(target=DomainData.excl2Tbl, args=(self, savedTable))
        p.start()
        p.join()
        arcpy.AddMessage("process ended")

    def excl2Tbl(self, tbl):
        arcpy.ExcelToTable_conversion(self.wkbkName, tbl)

if __name__ == "__main__":
    #do stuff
    sprFA_dom.createPointDomain()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
    #continue with the script‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This is the error that comes back:

pickle.PicklingError: Can't pickle 'geoprocessing server result object' object: <geoprocessing server result object object at 0x1E415BA8>

I get that geoprocessing objects of any sort aren't pickleable, but I don't need to do anything with the object created; I don't need to return it. How do I get the process to just ignore it?

Re: Multiprocessing help

MKF62 — Sun, 12 Dec 2021 06:37:55 GMT

Apparently multiprocessing doesn't appreciate class objects (e.g. "self") as arguments, so I needed to make my function global instead of just within the scope of the class that it really applies too. Then I was able to pass a property of self in-place of self. Perhaps there is another way to do this where you can use self as an argument and get things to work how I originally wanted them to, but this was the only solution I was able to make work. Here's the final code:

import arcpy, multiprocessing, os

####################
# Class Definitions
####################
class DomainData(object):
    '''This is the base class for all domain data'''
    def __init__(self, folderPath, gdbPath, dbCon):
        #properties

    def createDomain(self, wsRows, tblName):
        #Make Excel file
        #Save the file
        wb.save(self.wkbkName)
        savedTable = os.path.join(self.gdb, tblName)
        #Convert the excel file to an ArcMap table.
        p = multiprocessing.Process(target=excl2Tbl, args=(self.wkbkName, savedTable))
        p.start()
        p.join()
        arcpy.AddMessage("process ended")
        #do more stuff

##############################
# Global function definitions
##############################
def excl2Tbl(file, tbl):
    '''This function converts and excel file to an ArcMap table'''
    arcpy.ExcelToTable_conversion(file, tbl)

############
# Main code
############
if __name__ == "__main__":
    #do stuff
    sprFA_dom.createPointDomain()
    #continue with the script‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

topic Re: Multiprocessing help in Python Questions

Multiprocessing help

Re: Multiprocessing help