Multiprocessing viewsheds (easy)

Question

Hi there, 
I have been searching the forum for this but eventhogh it seems that the solution should be quite easy to come up with, I couldn't find one.

My system: Arcgis 10.1 Sp1, Python 2.7.2, I7 4770k 3,5 ghz, 16gb ddr3 ram, ssd drive

So here is my (simple) problem: I want to take advantage of the multiprocessing module to speed up viewshed calculations

In order to get to know this function I set up a folder containung 156 shapefiles each containing one single point. No I want to calculate viewsheds for each of these points using the multiprocessing module. It starts out fine: all cores are in use up to 100% and the right viewsheds are being calculated. but then it fails telling me either that "viewshe_demt1" already exists or that a second person or application is using this folder. Below are the error messages and my current code. I discovered that the code is producing folders called "viewshe_demtX" in the "scriptx"-folder which I don't really understand, because I specified the folder for the output of the viewshed analysis to be "scriptxy"

1) ExecuteError: ERROR 010429: "Error in GRID IO: CellLyrCreateInternal: Grid c:\studium\project\scriptx\viewshe_demt1 already exists."
2) (Couldn't find the error-code): "Another person or application is accessing this directory"
import arcinfo, os, re, multiprocessing
import arcpy
from arcpy import env
from arcpy.sa import *

arcpy.CheckOutExtension("Spatial")
env.workspace = "C:/studium/project/scriptx"
arcpy.env.overwriteOutput ='True'

arcpy.CreateFolder_management("C:/studium/project", "scriptxy")

def update_shapefiles(shapefile):
    '''Worker function'''
    outViewshed = Viewshed("C:/studium/00_Master/clean/demtora_okay1", shapefile, 2, "CURVED_EARTH", 0.15)
    mystring = "C:/studium/project/scriptxy/"+shapefile[26:37]
    outViewshed.save(mystring)
        
# End update_shapefiles
def main():
    ''' Create a pool class and run the jobs.'''
    arcpy.env.workspace = "C:/studium/project/scriptx"
    print arcpy.env.workspace
    fcs = arcpy.ListFeatureClasses('turbine*')
    fc_list = [os.path.join("C:/studium/project/scriptx", fc) for fc in fcs]
    print fc_list
    pool = multiprocessing.Pool()
    pool.map(update_shapefiles, fc_list)

# Synchronize the main process with the job processes to
    # ensure proper cleanup.
    pool.close()
    pool.join()
    # End main

if __name__ == '__main__':
    main()

I basically used the first code example from http://blogs.esri.com/esri/arcgis/2012/09/26/distributed-processing-with-arcgis-part-1/ and modified it (obviously poorly) to produce viewsheds. My normal Viewshed-analysis takes about 8 hours for 156 viewsheds and the CPU utilization never exceeds 12%. 
I think the problem with the new code has something to do with my file-naming and is otherwise fine. 
please help me get it to work properly.
any advise is welcome

StacyRendall1 · Answer

update: well, it seems like everytime one process (from the 8 different which are running simmultainiously) sets the environment extent to a certain extent, another process that is also running through the script uses this "global" extent and tries to create the next multiple bufferring with it. I'm not too sure about it, because stuff like this makes my brain hurt.
I tried to find a solution in which not the "global" extent (env.extent) is used, but one directly for the raster of the viewshed analysis. But all my efforts failed so far. Any ideas?

OK. Good job working it out! It is really useful to be able to do that.

Env stands for environment, the things you set in there will affect all Arc tools. It is called a 'global' variable, as any tool running will check what it is when it needs to. This can be quite problematic when parallelising problems, as if 8 worker processes are running near simultaneously the value will get changed 8 times, and you can't really guarantee what the value will be when accessed later on by each process. The same is true for the scratch workspace - this should not be in the worker process code - if you need to, make your own temporary folders/GDBs. The moral of the story is that you cannot reliably set any environment variables in your worker process.

The only thing you can do is what you were trying to do, somehow define the extent so that it applies only to the files being used by the worker process. To do this you need to understand the tools really well... I have no experience with these tools, sorry...

I also notice that there is a lot of code in your worker process (~23 functional lines) this is way too much. It should be stripped back to the bare minimum, only the most CPU intensive parts and the absolutely essential other components to support them. What I want you to do is run your worker process sequentially with timers around every operation. This will be a pain, but is essential to work out what is the actual thing you need to parallelise. You can do it like this:tS = time.clock()
### arcpy.SomeOperation()
arcpy.AddMessage('SomeOperation took: %.2f seconds' % (time.clock() - tS))

When you read my blog, did you work through all three posts on multiprocessing? I tried to make it describe a simple and clear process for getting a general problem to parallelise, but it could still need some work to increase clarity. I suggest you do each one in order and copy any code across by manually typing it - this really helps lock it in. If you are still getting stuck with the posts, please let me know what is confusing and I will try to improve them!

You also mention that apply_async doesn't work for you - can you provide any more information?

nb · Answer

okay I debugged it step by step and finally found out whats causing this strange behaviour: 
The multiple buffer rings (as well as the normal buffer)  work fine! hey buffer all the featureclasses. When I set arcpy.env.extent = buffVS

it somehow deletes my buffer feature class.
I checked this phenomenon 5 times to be extra sure that this is real. I edited the script back and forth and only this code causes the error. It still can not have to do anything with my point data, because, when I don't execute the line with the arcpy.env.extent = buffVS

it calculates the buffer-files and they ALL have a valid extent. I  triplechecked.

I'm pretty confused by this behaviour of arcpy.env.extent = path. Why does it kill my buffer feature class and why does it work with all the other feature classes????

update: well, it seems like everytime one process (from the 8 different which are running simmultainiously) sets the environment extent to a certain extent, another process that is also running through the script uses this "global" extent and tries to create the next multiple bufferring with it. I'm not too sure about it, because stuff like this makes my brain hurt.
I tried to find a solution in which not the "global" extent (env.extent) is used, but one directly for the raster of the viewshed analysis. But all my efforts failed so far. Any ideas?

nb · Answer

for some crazy reason the buffer analysis as well as the multiple ring buffer analysis skip certain points. If I manually delete these points the error passes to the next point.

if I use multiprocessing the buffer and multibufferrings of the singlepoints 13, 55, 75, 99, 105, 110, 115 from 156 single point features are not created
if I use single core processing, so the normal stuff without moduls and pool.map than ist is just the 13 and the 99 that are skipped.
I find this so frustrating...especially because it doesn't happen if I buffer or multibuffer the whole multipoint shapefile.