How can I start a script or model on the condition that a previous step has completed normally.

[PaulHuffman]

I was processing some big rasters from Lidar data last week.  There were five basic steps I wanted to complete: 1. Terrain to raster 2. Fill. 3. Flow Direction, 4. Flow Accumulation, 5. A Con to select the higher values of the Flow Accumulation to get to stream lines.    While step 2 was running, I started thinking "Hey, this is going to take a while.  I could put steps 3,4, and 5 in a script or model, and get them ready to run unattended once this Fill completes."  But the Fill was still running when I went home.  This got me wishing that I could launch the script running the remaining steps, but make the script wait until the Fill completed.  Like I used to do in Unix environments.

I installed the wmi module for python, http://stackoverflow.com/questions/1632234/list-running-processes-on-64-bit-windows 

but when I ran it, my process list was empty.  They say it runs the same for Windows 7 64 like I am running, but I didn't figure this out.

I took a look at wmic  http://stackoverflow.com/questions/16326529/python-get-process-names-cpu-mem-usage-and-peak-mem-usag...

. I could get a list of my processes at cmd,  so it will work in python,  but I didn't figure out a way recognize my Fill process or a way to test for the absence of my Fill process.  I suppose the way to do it is to have the script run in circles until the process is not found, then let the script continue, like I did once with Unix shell scripts. Does anyone have an example of how to do this in Windows.

[DarrenWiens2]

A python script runs one line at a time. Once the Fill tool call has completed normally, it moves onto the next line, where for you it would call the Flow Accumulation tool. Please let me know what I'm missing. Are you somehow running this independent of python?

[PaulHuffman]

Sorry if I didn't make this clear.  Step one was finished, I started step two in the late afternoon and was surprised that it was taking so long.  I wanted to put steps 3, 4, 5 into a script,  get it in the que somehow so it would wait until the Fill operation completed before starting in on steps 3, 4 , and 5, sometime during the night, so it be all finished in the morning. 

I had this other idea today that rather than reach into the OS to see if the process was still running,  maybe the test could be an arcpy Exists.  However, is it possible that a grid can pass the Exist test when it's not really finished?  I have a Fill raster now that was able to be added to ArcMap, but will not draw, and it seems like the script that created it is still running.

[PaulHuffman]

An associate pointed out to me that if you have installed the 64 Bit Background Geoprocessing Tools,  the Fill tool is running asynchronously in the background.  You can start another geoprocessing tool or script from the ArcMap or ArcCatalog session you used to do the Fill, and it will wait to run until your Fill process has finished.  But in my case, I haven't turned on background processing, and,  besides, I had started the first script from IDLE, not the python window or tool within ArcMap. 

Rather than asking the OS about processes, I think all I need to do is use arcpy.Exists by adding something like the following to the top of my second script:

import arcpy
import time
from arcpy import env
from arcpy.sa import *
arcpy.CheckOutExtension("Spatial")

env.workspace = r"G:\elevation\lidar_2005\ThreeWay\ThreeWayTerrians.gdb"
while not arcpy.Exists("Fill_TWtoPom12"):
          time.sleep(10)
#rest of the script

......
......

But I wonder if Exists might jump the gun and evaluate as true before the raster is fully complete. I think I saw an example yesterday of a raster that showed up in Catalog, was able to be added to a map, but had map draw errors because it wasn't completely done.  Maybe I can test this later today.

[PaulHuffman]

I was able to test this, and it looks like Exists evaluates as true before the raster is completed.  I ran  a little script when the input raster, strm2M, was about 38% complete -

import arcpy
import time
from arcpy import env
from arcpy.sa import *
#arcpy.CheckOutExtension("3D")
arcpy.CheckOutExtension("Spatial")

env.workspace = r"G:\elevation\lidar_2005\ThreeWay\ThreeWayTerrians.gdb"
while not arcpy.Exists("strm2M"):
          time.sleep(10)
          #print "File not found"
print "File found"
arcpy.RasterToPolyline_conversion("strm2M", "strm2Mline", "ZERO",
                                   50, "SIMPLIFY")

This script bombed at line 14 with a "cannot acquire a lock" and "Error 010213: Error in reading raster Band_1". When I ran the script after the con tool that was making strm2M completed,  it hit an error that "strm2Mline exists. Can't overwrite".  So the previous run created a new feature class but wasn't able to put any features in it. The attribute table had no records.  When I deleted the interrupted feature class  strm2Mline, the script ran successfully.

Looks like arcpy.Exists doesn't work for this. I need something like arcpy.ReallyCompletedReleasedLockThisTime..

[DarrenWiens2]

You may be able to hack together some code using the following logic (found here). It should produce an infinite loop calling a function until there is no error, which you may want to just stop and consider.

result = None
while result is None:
  try:
      result = get_data(...)
  except:
      pass

[PaulHuffman]

I hope to try this later today.  I'm a bit concerned that a solution like this could gobble up CPU while it runs and fails.  As one of the commenters on stackoverflow offers: "But any proposed answer should be safe, or at least note the pitfalls. This does not offer protection against 100% CPU consumption and endangers future readers. –  Brad Koch May 30 '14 at 19:03"

Putting a sleep inside the while or try might help reduce the CPU use, but I don't know for sure.  I'm guessing sleep uses less than attempting an operation on a big raster and failing. 

Also needed will be env.overwriteOutput, since I have seen above that a tool can create an output file even though the input incomplete and the tool going to therefore fail. 

[PaulHuffman]

I think this worked for me.  I launched this script from IDLE, waited a couple minutes, the IDLE shell window showed a single set of >>> after the restart and wouldn't advance to a blank line, just beeped at me.  CPU usage clicked along at 6 to 14%.  Then I created the input raster by copying SimHAWS2000 to SimHAWS2000_1, and in a couple minutes both the output rasters came out. 

# Import arcpy module
import arcpy
import time
from arcpy import env  
from arcpy.sa import * 

# Check out any necessary licenses
arcpy.CheckOutExtension("spatial")

arcpy.env.overwriteOutput = True

# Local variables:
SimHAWS2000_1 = "G:\\elevation\\lidar_2005\\ThreeWay\\ThreeWayTerrians.gdb\\SimHAWS2000_1"
Input_true_raster_or_constant_value = "8"
Input_true_raster_or_constant_value__2_ = "-8"
Con_SimHAWS20001 = "G:\\elevation\\lidar_2005\\ThreeWay\\ThreeWayTerrians.gdb\\Con_SimHAWS20001"
SimHAWS2kCon = "G:\\elevation\\lidar_2005\\ThreeWay\\ThreeWayTerrians.gdb\\SimHAWS2kCon"

result = None
while result is None:
    try:
        time.sleep(60)
        # Process: Con
        arcpy.gp.Con_sa(SimHAWS2000_1, Input_true_raster_or_constant_value, Con_SimHAWS20001, SimHAWS2000_1, "VALUE >= 8")

        # Process: Con (2)
        arcpy.gp.Con_sa(Con_SimHAWS20001, Input_true_raster_or_constant_value__2_, SimHAWS2kCon, Con_SimHAWS20001, "VALUE <= -8")
        result = "Both Cons executed"
    except:
        pass