multiprocessing Attribute Error

4939
4
12-07-2011 10:38 PM
JohanL
by
New Contributor
Has anyone tried to implement multiprocessing into their scripts? I've attempted to use the suggestion from http://blogs.esri.com/Dev/blogs/geoprocessing/archive/2011/08/29/Multiprocessing.aspx with raster processes but with no success i.e.


import os, multiprocessing, arcpy 

def project(raster):
        arcpy.ProjectRaster_management(raster...) 

def main():
        workspace = r'c:\somepath'
        arcpy.env.workspace = workspace
        fcs = arcpy.ListRasters('*')
        fc_list = [os.path.join(workspace, fc) for fc in fcs]
        pool = multiprocessing.Pool()
        pool.map(project, fc_list)
        pool.close()
        pool.join()

if __name__ == '__main__':    
        main()


However in the command line this gives me the error


File "C:\Python26\ArcGIS10.0\lib\multiprocessing\process.py", line 232, in _bootstrap
     self.run()
File "C:\Python26\ArcGIS10.0\lib\multiprocessing\process.py", line 88, in run
     self._target(*self._args, **self._kwargs)
File "C:\Python26\ArcGIS10.0\lib\multiprocessing\pool.py", line 57, in worker
     task = get() File "C:\Python26\ArcGIS10.0\lib\multiprocessing\queues.py", line 352, in get return recv()
AttributeError: 'module' object has no attribute'project'


I've been trying to work around the problem with parallel python without success and I cannot figure out how my code differs from the original.

Bjebn
Tags (2)
0 Kudos
4 Replies
StacyRendall1
Occasional Contributor III
Bjorn,

Your code looks a little more complicated than it needs to be, however it ran OK when I simply copied what you had (and changed the contents of project() to some generic math).

When I tested it with actual ProjectRaster I realised that you don�??t specify an output location, and if you are using pool.map() you can only pass a single argument to the function. Consequently, I had to change the code around to allow two things (input and output paths) to be passed to project()�?� To do this I used the job server notation that was in my multiprocessing post.

I also took out the unnecessary function main() that you had (from copying the ESRI example, I presume) �?? scripted things will run just fine from within the if __name__ == �??__main__�??: statement. This code works perfectly for me (tested with 5 reasonably large rasters):

import os, multiprocessing, arcpy
def project(rasterin, rasterout):
    arcpy.ProjectRaster_management(rasterin, rasterout,#�?�
    # just return something; will print once done
    return �??converted: %s and made output: %s�?? % (rasterin,rasterout)

if __name__ == �??__main__�??:
    workspace = r�??d:\Temp\dems�??
    arcpy.env.workspace = workspace
    fcs = arcpy.ListRasters(�??*�??)

    pool = multiprocessing.Pool()
    jobs = []

    # iterate through features, add input and output paths
    for fc in fcs:
        rIn = os.path.join(workspace, fc)
        rOut = os.path.join(r�??d:\Temp\demsOut�??, fc)
        jobs.append(pool.apply_async(project, (rIn, rOut)))

    for job in jobs:
    print job.get()
0 Kudos
JohanL
by
New Contributor
Hi, Stacy

Thanks once again for your help... Out of curiosity have you ever run into the problem where the first batch of multiprocessing jobs are completed and the next set of batches are about to be loaded but fail at the first geoprocessing? Namely I receive the following error...


ERROR 010302: Unable to create the output raster: C:\Data\sr6f8~1\FocalSt_srtm1
ERROR 010067: Error in executing grid expression.
Failed to execute (FocalStatistics).


I believe FocalSt_srtm1 is a temporary file created by arcpy for Focal statistics however I am not sure why a seperate folder is created with the strange characters 'sr6f8~1' (perhaps an os or arcpy thing?) It contains the FocalSt_srtm1 file and files named dblbnd.adf, hdr.adf, w001001.adf and w001001x.adf.

I have also started to run my programs through python.exe rather than the pythonw.exe window as pythonw.exe does not appear to close the program after execution has finished as apparently due to its design.

Kind Regards,
Bjorn

ps - When I study the performance more it appears as though certain processing are not being performed by several cores simultaneously (with CPU down to around 10%) which might be a limitation of arcpy?
0 Kudos
JohanL
by
New Contributor
***UPDATE***

I continue to receive the ERROR 101302/ERROR 10067 where FocalSt_strm1 is trying to be created but cannot. The FocalSt_srtm1 is a file created in association with focal statistics analyses and I do not know if that name can be changed at all.

One interesting note is that the strange sr.. folders that are generated are only created for those processes that are queued in the jobs list??? Those that have been processed properly (i.e. the first batch) do not have them.

Any help or support documents from ESRI on the subject would be most apprecaited.

Sicnerely,
Bjorn
0 Kudos
JasonTaylor
New Contributor
I'm not sure about your particular issue, but I did have the same combo of error messages 101302 and 10067. It seemed to me like Arc was having permission issues and I looked to see what permissions were set to the folder I was outputting to. Although I have most permissions on my machine, the folder was listed by both windows explorer and arc catalog as being read-only in the folder properties. This was demonstrated by a greyed out check mark. Greyed out usually means I can't access the change, however I was able to uncheck the read-only box. My raster tools are now able to output to that directory. I was getting the errors while try to recondition a DEM using ArcHydro.

It makes no sense to me why this was set up as read only, or why the read-only box was greyed out, but this seems to have solved my problem. As another note, the box seems to be re checking itself after some activity. I need to go in periodically and un-check the box if I want to run raster tools in ArcHydro.
0 Kudos