Geoprocessing Service Output Directory

[KyttMacManus]

I am running a GP Service and want to save my outputs to a local File GDB. I have tried hard coding the output file paths as a parameter, but it seems they are automatically replaced by the Scratch.GDB in an individual job directory.

Is it possible to save the outputs from a GP Service to a local folder? I have already registered the folder with the data store and that seems to be working alright.

Thanks for your help.

Kytt MacManus
Geographic Information Specialist
Columbia University CIESIN

[KevinHibma]

It is possible, but you have to set it up in a certain way.

Can I ask why you want to do this? From my experience I've seen nothing but trouble when trying to write the result to somewhere other than the scratch folders. You're on your own in making sure the output names are unique or in the very least can overwrite existing output. And there is also the question if you have 2 instances running at the same time: They're both going to try and write the output to the same spot/same name.

Anyways, if you're sure this is what you want to do, this basic script demonstrates how to do it. The point is having a variable point to a folder and that folder being registered with the data store. During the publishing process that folder is found in the datastore, thus data that is inside that folder is NOT copied, nor updated to scratchFolder/scratchGDB inside the script.

import arcpy, os

arcpy.env.overwriteOutput = True

inPts = arcpy.GetParameterAsText(0)

inDistance = arcpy.GetParameterAsText(1)

outputFolder = r"c:\gpServices\writeToFolder"  #this needs a matching entry in the datastore.
#If you have the output param set to required, make sure the path/name in the dialog matches the path set inside the script
outputFC = os.path.join(outputFolder, r"output.gdb\output2") 

arcpy.Buffer_analysis(inPts, outputFC, inDistance)

Goodluck.

[KyttMacManus]

Thanks very much for your reply Kevin.

We are seeking to set up an asynchronous WPS for the creation of gridded demographic variables as part of the Gridded Population of the World Version 4 ( see http://sedac.ciesin.columbia.edu/data/collection/gpw-v3)

At the national level we have a high granularity of variable (Population by sex by age group) etc...  Many more variables than we could possibly grid ourselves ahead of time for distribution.

Instead we are developing a system where user requests trigger the gridding algorithm assuming that we have not already produced the file ahead of time.

Once a file has been requested and produced we would like to cache it for future requests.  That is why we need to be able to copy it to a local or UNC path.

I will test out this code and let you know how it goes.

Thanks very much again.

Kytt MacManus
Geographic Information Specialist
Columbia University CIESIN

[KevinHibma]

Ok.... so you could possibly let your script do all "intermediate" processing in the scratch space and then have one copy at the end to persist your output to a known location.
This would be much "easier/safer" then doing everything in a local, non-server managed directory.

The code in should be straight forward:
-do intermediate processing (file raster output saved to scratchFolder on the server)
-check directory to copy file to (reading list of files)
-file name = last file +1 (or some idea to keep it unique and incremented - maybe a datestamp on the file name?)
-copy file

Of course there is still the chance of 2 or more service executions trying to write at the same time, but the chance of this depends mostly on how often the service will be executed.

[KyttMacManus]

Hi Kevin,

Sorry that it has taken me so long to get back to this, you know how it goes.

I am still having trouble.  I have implemented my code with a directory which is registered in the Data Store as a variable.  When I execute on the server it finishes fine, but then I am unable to locate the output in either the scratchGDB or the local GDB I specify.

This is the code I am using:

# Import Python Libraries
import arcpy, os, string
from arcpy import env

# Set Overwrite Output Environment
env.overwriteOutput = True

# Define Output Folder
outputFolder = r"E:\arcserver\data\tooldata"

# helper method to check if a field exists in a fc
def check_for_field(featClass,fieldName):
    hasField = 0
    desc = arcpy.Describe(featClass)
    fields = desc.fields
    for field in fields:
        # check without case as ArcGIS is not case sensitive
        if field.name.upper() == fieldName:
            hasField = 1    
    return hasField

# Define inputs
# Input ISOCODE
fcString = arcpy.GetParameterAsText(0)
# Project Data
# Input Fishnet
fish = outputFolder + os.path.sep + fcString + ".gdb" + os.path.sep + fcString + "_fishnet_full_attributes"
# Coordinate System
wgs84 = arcpy.SpatialReference(4326)
# Describe Fish
desc = arcpy.Describe(fish)
# Calculate Raster Extent
extent = desc.Extent
xmin = int(round(extent.XMin - .5))
xmax = int(round(extent.XMax + .5))
ymin = int(round(extent.YMin - .5))
ymax = int(round(extent.YMax + .5))

# Define Gridding Variables
gridFieldsWildCard = arcpy.GetParameter(1)#"*TOTPOPBT_2010*M"
gridFields = arcpy.ListFields(fish,gridFieldsWildCard)

# Lines per degree, determines the output resolution 120 = 30 arc-seconds resolution
# 1 degree divided into 120 parts is 30 seconds
linespd = 120 ## Update As Needed
cellSize = 1.0 / linespd
# Output the gridded count rasters
for field in gridFields:
    gridField = field.name
    arcpy.AddMessage("The field to be gridded is " + gridField)
    # Output Grids
    outPopGrid =  outputFolder + os.path.sep + fcString + "_grids.gdb" + os.path.sep + fcString + "_" + gridField
    arcpy.env.extent = arcpy.Extent(xmin,ymin,xmax,ymax)
    arcpy.env.outputCoordinateSystem = wgs84
    arcpy.env.cellSize = cellSize
    arcpy.AddMessage("The extent is " + str(arcpy.env.extent))
    if arcpy.Exists(outPopGrid):
        arcpy.AddMessage(outPopGrid + " exists")
    else:
        try:
            arcpy.PolygonToRaster_conversion(fish,gridField,outPopGrid,'CELL_CENTER','#',cellSize)
            arcpy.AddMessage("Created " + outPopGrid)
        except:
            arcpy.AddMessage(arcpy.GetMessages())

Any advice you could provide would be very helpful.

Thanks so much,
Kytt

[KyttMacManus]

UPDATE

I was able to successfully save to a local GDB registered with my DataStore if I first wrote the grids to the scratchGDB and then copied them to the local GDB.

It would still be great to have an explanation for why this is if you are able to reproduce the problem based on the script above.

Thanks so much again for your help Kevin.

-Kytt