chriskrow

Multiprocessing using arcpy & spatial analyst (solar analyst tool)

Discussion created by chriskrow on Apr 9, 2015
Latest reply on Apr 3, 2016 by Dan_Patterson

I need help getting multiprocessing to work with the spatial analyst tool - Solar Analyst.

 

 

I have tested the multiprocessing bit and it works fine. This sample is passing it a file containing 2 lines used as parameters that get parsed and fed to the main function (final) as vars to the Solar Analyst tool. The function works on its own, but when altogether with the multiprocessing code I get the traceback below.

 

 

I think that the fact the raster DEM is trying to be accessed by multiple child processes is what is tripping the traceback error, as I get the error after some time, close to what would be the second run from the pool. Perhaps it is because I do not have unique scratch workspaces for reach child process, as is mentioned in the following links: link , link. Again, if this is the case I need help implementing a unique workspace for each child process.

 

 

However, as suggested by a colleague I have successfully run the Solar Analyst tool using the same input raster on two instances of ArcMap being open, at the same time. So I am not convinced it is having a hard time accessing the raster because the other child process is as well. Thoughts?

 

 

In general, please feel free to offer any pointers you might have for optimizing this code. Thank you kindly.

 

 

    import multiprocessing
    import time
    import arcpy
  
    def final(x):
        outputfile=open("outputfile.txt","a")
        starttime= time.time()
        # Check out any necessary licenses
        arcpy.CheckOutExtension("Spatial")
        arcpy.env.workspace = r"C:\data\LiDAR\working\V2\FINAL_TEST\final_test"
        arcpy.env.overwriteOutput = True
  
        # Parse params
        the_split= x.split(",")
        DOY = the_split[0]
        k_daily = the_split[1]
        diffuse = the_split[2]
  
  
        # Set local variables
        inRaster = r"C:\data\LiDAR\working\V2\best_dems_05_1\oak_idwnat_flt.tif"
        latitude = 48.4222
        skySize = 400
        timeConfig = arcpy.sa.TimeWithinDay(DOY,0, 24)
        dayInterval = 14
        hourInterval = 0.5
        zFactor = 1
        calcDirections = 32
        zenithDivisions = 16
        azimuthDivisions = 16
        diffuseProp = diffuse
        transmittivity = k_daily
        outDirectRad = ""
        outDiffuseRad = ""
        outDirectDur = ""
  
        # Execute AreaSolarRadiation
        outGlobalRad = arcpy.sa.AreaSolarRadiation(inRaster, latitude, skySize, timeConfig,
          dayInterval, hourInterval, "NOINTERVAL", zFactor, "FROM_DEM",
          calcDirections, zenithDivisions, azimuthDivisions, "UNIFORM_SKY",
      diffuseProp, transmittivity, outDirectRad, outDiffuseRad, outDirectDur)
  
        # Save the output
        outGlobalRad.save(r"C:\data\LiDAR\working\V2\FINAL_TEST\output\global_"+str(DOY))
        end_time = time.time()
        outputfile.write("batch job" + str(DOY) + "ended well")
        outputfile.write(" %d " % (end_time-starttime) )
        outputfile.close()
    #To test this script only pass the below
  
    #if __name__ == "__main__":
    # final("266,0.4269347,0.771440152")
  
    if __name__=='__main__':

       start= time.time()
  
       filename = "k_daily_diffuse_2.txt"
       readfile = open(filename, "r")
       lines_from_filename = []
       for i in readfile:
           lines_from_filename.append(i)
       p = multiprocessing.Pool(2)
       p.map(final, lines_from_filename)
       end = time.time()
       print "batch job ended well"
       print end-start
  
       p.close()
       p.join()

 

----------------------------------------OUTPUT BELOW----------------------------------------

 

 

    Python 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] on win32 Type "copyright", "credits" or "license()" for more information.
    >>> ================================ RESTART ================================
    >>>
  
    Traceback (most recent call last): 
    File "C:\data\LiDAR\working\V2\FINAL_TEST\multi_test_FINAL.py", line 63, in <module>
            p.map(final, lines_from_filename) 
    File "C:\Python27\ArcGISx6410.2\lib\multiprocessing\pool.py", line 250, in map
            return self.map_async(func, iterable, chunksize).get() 
    File "C:\Python27\ArcGISx6410.2\lib\multiprocessing\pool.py", line 554, in get
            raise self._value ExecuteError:
    ERROR 999999: Error executing function.
    Failed to open raster dataset
    Failed to execute (AreaSolarRadiation).

 

These sites report the same traceback errors, if they offer any clues:

[GeoNet from 2010], [GeoNet from 2014], [GIS StackExchange]

 

The community deserves an answer to this longstanding problem.

This question was posted here hoping an answer might come from a source that did not see the same question at this site, as Dan Patterson is keen to point out. Please keep any comments to the point of this post: provide constructive feedback towards an end to the problem. Thanks.

Outcomes