need to clear scratch memory cache

JoannaWhittier — Thu, 02 Feb 2012 13:31:59 GMT

I have looked at several posts here and on other sites but have not found a solution to the problem I am having with this python script. In this script I use 2 nested 'for' loops to extract a series of variables by month (month = 0 - 11) from 5 netcdf files and then mosaic together each extracted variable. The script runs but after processing about 3 variables, I get the following error statement:

ExecuteError: ERROR 999999: Error executing function. Failed to copy raster dataset Failed to execute (MosaicToNewRaster).

My suspicion is that the cache memory has been exceeded. I tried using arcpy.Delete_management(SCRATCH) but then received the error that the scratch space no longer existed after only one variable was processed. Maybe I am putting that statement in the wrong place? Any other suggestions?

# Import arcpy module
import arcpy
import os

# Set Geoprocessing environments
arcpy.env.scratchWorkspace = "C:\\WORKSPACE\\Process_4"
SCRATCH = arcpy.env.scratchWorkspace


# Script arguments
ENA = arcpy.GetParameterAsText(0)
if ENA == '#' or not ENA:
    ENA = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_ena_eh5_2030-2034_avg_trim_v4.nc" # provide a default value if unspecified

NRM = arcpy.GetParameterAsText(1)
if NRM == '#' or not NRM:
    NRM = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_nrm_eh5_2030-2034_avg_trim_v4.nc" # provide a default value if unspecified

PNW = arcpy.GetParameterAsText(2)
if PNW == '#' or not PNW:
    PNW = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_pnw_eh5_2030-2034_avg_trim_v4.nc" # provide a default value if unspecified

PSW = arcpy.GetParameterAsText(3)
if PSW == '#' or not PSW:
    PSW = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_psw_eh5_2030-2034_avg_trim_v4.nc" # provide a default value if unspecified

SRM = arcpy.GetParameterAsText(4)
if SRM == '#' or not SRM:
    SRM = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_srm_eh5_2030-2034_avg_trim_v4.nc" # provide a default value if unspecified


Output_Location = arcpy.GetParameterAsText(5)
if Output_Location == '#' or not Output_Location:
    Output_Location = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\summary\\" # provide a default value if unspecified
 

#Variable list
for variable in ['T33', 'TA', 'TAMAX', 'TAMIN', 'TOTRNF', 'ET', 'RT','RNFS']:
    

#    SCRATCH = arcpy.env.scratchWorkspace               I attempted to place SCRATCH within the loop thinking that it would be recreated.  Nope.

    #Month Iteration
    for month in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]:
        ENA_Layer = SCRATCH +"\\ENA"+variable+str(month)
        NRM_Layer = SCRATCH +"\\NRM"+variable+str(month)
        PNW_Layer = SCRATCH +"\\PNW"+variable+str(month)
        PSW_Layer = SCRATCH +"\\PSW"+variable+str(month)
        SRM_Layer = SCRATCH +"\\SRM"+variable+str(month)

        # Process: Make NetCDF Raster Layer (169)
        arcpy.MakeNetCDFRasterLayer_md(ENA, variable, "x", "y", ENA_Layer, "", "time "+ str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (157)
        arcpy.MakeNetCDFRasterLayer_md(NRM, variable, "x", "y", NRM_Layer, "", "time "+ str(month), "BY_INDEX")

  # Process: Make NetCDF Raster Layer (145)
        arcpy.MakeNetCDFRasterLayer_md(PNW, variable, "x", "y", PNW_Layer, "", "time "+ str(month), "BY_INDEX")

  # Process: Make NetCDF Raster Layer (133)
        arcpy.MakeNetCDFRasterLayer_md(PSW, variable, "x", "y", PSW_Layer, "", "time "+ str(month), "BY_INDEX") 

  # Process: Make NetCDF Raster Layer (121)
        arcpy.MakeNetCDFRasterLayer_md(SRM, variable, "x", "y", SRM_Layer, "", "time "+ str(month), "BY_INDEX")  
  
  
        # Process: Mosaic To New Raster (25)
        arcpy.MosaicToNewRaster_management("\"" + ENA_Layer +";"+ NRM_Layer +";"+ PNW_Layer +";"+ PSW_Layer +";"+ SRM_Layer +"\"", Output_Location, variable+"_"+str(month)+"2032", "PROJCS['WGS_1984_Lambert_Conformal_Conic',GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Lambert_Conformal_Conic'],PARAMETER['false_easting',0.0],PARAMETER['false_northing',0.0],PARAMETER['central_meridian',-102.3000030517578],PARAMETER['standard_parallel_1',30.0],PARAMETER['standard_parallel_2',60.0],PARAMETER['latitude_of_origin',52.0],UNIT['Kilometer',1000.0]]", "", "1", "1", "MEAN", "FIRST")

   arcpy.Delete_management(SCRATCH)

Re: need to clear scratch memory cache

HenryColgate — Mon, 06 Feb 2012 02:07:44 GMT

I don't have a specific solution to your problem as specific code but I have a general suggestion for working with NetCDF.

Having said that I have made similar code for NetCDF data but have taken a different approach structurally with my code and have not run into any problems.

Essentially I have split this code into three sections that are tied together with my main function but can be run consecutively as seperate functions.

Firstly I extract the data to individual rasters. Then there is a bit of metadata stuff that I retrieve from external sources. Then I add the info to a mosaic dataset.

If you split your code into parts as opposed to dealing with it in nested loops I think you are less likely to run into memory problems. I also find code like this easier to debug but that is just personal preference.

topic Re: need to clear scratch memory cache in Python Questions

need to clear scratch memory cache

Re: need to clear scratch memory cache