topic Re: Uploading and using data to a geoprocessing service in Geoprocessing Questions

Uploading and using data to a geoprocessing service

TomSchuller — Thu, 20 Dec 2012 03:32:54 GMT

Hy,
I'm trying to build a simple geoprocessing service which should buffer an uploaded shapefile.

This is the scenario:
1) uploading shapefile to geoprocessing service
2) buffering the shapefile with a given parameter
3) returning the buffered shapefile

The buffering is not the problem, but the handling of the upload and reusing it in the model or python part.
As far as I read, I should have to use a Datafile parameter for upload, but I'm always getting this error while running it:
ExecuteError: Failed to execute. Parameters are not valid. ERROR 000735: input: Value is required Failed to execute

Can anybody help me to use uploads on geoprocessing services?

Thanks,
Tom

Re: Uploading and using data to a geoprocessing service

KevinHibma — Thu, 20 Dec 2012 15:14:13 GMT

Upload to a GP Service is sort of a multistep process.

You're correct, you do want to use a "file" (GPDataFile) as the input parameter.
If you're using Destkop to consume the service, desktop will send the file to the server.
If you're using a webapp, you're responsible for getting the file to the server. (Server at 10.1 has some upload capabilities in REST you can leverage. If using 10.0 or prior you'll need to devise a way to get the file to the server your self)

Once the file has been upload, because you're talking shape file, you'll have to unzip it. (shapefile being multiple files to make up a single one).
With it unzipped, you can then pass the file to your service.

Based on your error - your file didnt upload, or you never sent it. Whats the client to consume the GP Service - a web app?

Re: Uploading and using data to a geoprocessing service

TomSchuller — Fri, 21 Dec 2012 03:39:42 GMT

thanks for the quick answer.
Yes, I know it's a multistep process.

I have the problem that I don't see howto handle an uploaded file over gpservice in a model or python script.
Did you have an example of your "unzipModel" scenario, like:
user is uploading data to gpservice (10.1 upload capabilities in rest)
user is calling the unzipModel with the upload-id as parameter (over rest url)
the model is unzipping the zip file into some directory

I just need to see how I can use the uploaded file in a python script or model directly.

Thanks,
Tom

Re: Uploading and using data to a geoprocessing service

TomSchuller — Fri, 21 Dec 2012 03:44:16 GMT

thanks for the quick answer.
Yes, I know it's a multistep process.

I have the problem that I don't see howto handle an uploaded file over gpservice in a model or python script.
Did you have an example of your "unzipModel" scenario, like:
user is uploading data to gpservice (10.1 upload capabilities in rest)
user is calling the unzipModel with the upload-id as parameter (over rest url)
the model is unzipping the zip file into some directory

I just need to see how I can reuse the uploaded file in a python script or model directly.

Thanks,
Tom

Re: Uploading and using data to a geoprocessing service

LaurynasGedminas — Wed, 02 Jan 2013 12:03:24 GMT

Hello,
Having the similar issue here, I created python script to upload text files to SDE,
published the script as GP service and it works in ArcMap and ArcGISExplorer.
In Flexviewer added this GP service with input parameter as "datafile" and i cant upload the files

Thanks,

Laurynas

Re: Uploading and using data to a geoprocessing service

TomSchuller — Wed, 02 Jan 2013 12:46:43 GMT

Hy,
I didn't yet used it in the FlexViewer but only manually over rest.

But it should point you to the right direction:
enable "upload" on your geoprocessing service
upload your file to the gp-service and remember the upload-id
call your gp-service with the datafile paramater as the text below:

For GPDatafile parameter, you have to indicate this full json representation of the upload item:
parameter = {"itemID":"i80dfa12f-52ed-4841-94ff-37f9c3f5dd6f"}

Tom

Re: Uploading and using data to a geoprocessing service

LaurynasGedminas — Thu, 03 Jan 2013 11:12:27 GMT

Thanks Tom,
I guess I was not clear about my problem
i dont know why flexviewer doesnt display "upload" button on its widget when input data type is set to "datafile"

Laurynas

Re: Uploading and using data to a geoprocessing service

BjornSvensson — Tue, 15 Jan 2013 21:03:58 GMT

If ESRI will update the geoprocessing widget so it will take a local file, I, and many other users will be very happy!

Thanks for the suggestion to support local file uploads in the Geoprocessor widget. We will take a look at this for version 3.2 of the Flex Viewer. It would most likely just follow the steps outlined by Dasa in the thread you listed: http://forums.arcgis.com/threads/64114-Uploads-capability-on-Geoprocessor.

As for the ArcGIS API for Flex - since the Flash API already supports file upload, I don't think this is something we would add to the ArcGIS API for Flex.

Re: Uploading and using data to a geoprocessing service

meriyalootka — Fri, 25 Jan 2013 03:51:46 GMT

Hy
I have a model for uploading datafile, every thing is ok and user's zip file extract correctly. But I can not use extracted shapefile. Is there any py code for using feature class in a scratchfolder? Can I have this part of your model?

Hy,
I'm trying to build a simple geoprocessing service which should buffer an uploaded shapefile.

This is the scenario:
1) uploading shapefile to geoprocessing service
2) buffering the shapefile with a given parameter
3) returning the buffered shapefile

The buffering is not the problem, but the handling of the upload and reusing it in the model or python part.
As far as I read, I should have to use a Datafile parameter for upload, but I'm always getting this error while running it:
ExecuteError: Failed to execute. Parameters are not valid. ERROR 000735: input: Value is required Failed to execute

Can anybody help me to use uploads on geoprocessing services?

Thanks,
Tom

Re: Uploading and using data to a geoprocessing service

meriyalootka — Fri, 25 Jan 2013 03:53:21 GMT

The datafile parameter input for the geoprocessing widget in the Viewer for Flex does not allow the user to upload a file to the server from their local machine. It only accepts paths that begin with http:// or https://

I was under the impression that this was going to be fixed at the 3.1 release, but the widget still won't accept a local file.

This thread indicates all the pieces are available if you have the capability of creating your own custom widget:

http://forums.arcgis.com/threads/64114-Uploads-capability-on-Geoprocessor

There are many of us that cannot create custom widgets. If ESRI will update the geoprocessing widget so it will take a local file, I, and many other users will be very happy!

http://forums.arcgis.com/threads/54441-Uploading-to-SDE-from-Flex-Viewer
http://forums.arcgis.com/threads/41089-datafile-input-broken-in-Geoprocessing-widget
http://forums.arcgis.com/threads/36560-Geoprocessing-widget-has-me-stumped
http://forums.arcgis.com/threads/60510-Geoprocessing-with-File-Input

Re: Uploading and using data to a geoprocessing service

KevinHibma — Sun, 12 Dec 2021 03:20:51 GMT

The following code (10.1 + only) walks through a directory and finds FeatureClasses (this example assumes a folder of "zip" files).
It then copies each one into a fGDB. You can change what it does, but the logic in it should start you off.
You'll notice that a directory is created inside the scratchFolder (the variable "ZipFolder") - this is where the uploaded ZipFile is extracted to. Then the code looks in there.

import arcpy
import zipfile
import os

inFile = arcpy.GetParameterAsText(0) 

# Create a folder in the scratch directory to extract zip to
zipFolder = os.path.join(arcpy.env.scratchFolder, "zipContents")
os.mkdir(zipFolder)


# Extract the zip contents
zip2Extract = zipfile.ZipFile(inFile, 'r')
zip2Extract.extractall(zipFolder)
zip2Extract.close()

# Create a folder in the scratch directory to hold the fgdb which will be downloaded
fgdbFolder = os.path.join(arcpy.env.scratchFolder, "fgdbOutput")
os.mkdir(fgdbFolder)

# Work through all the FeatureClasses inside the extracted zip folder
for dirpath, dirnames, filenames in arcpy.da.Walk(zipFolder, datatype="FeatureClass"):
    
    for filename in filenames:
        
        # You could replace the code below here with your own code to do what you want....
        arcpy.AddMessage("Copying: {}".format(filename))
        
        # Strip .shp from the filename when merging shapefiles
        if filename.endswith("shp"):
            outFilename = filename[:-4]
        else:
            outFilename = filename

        # Copy each featureclass into the output.gdb
        arcpy.CopyFeatures_management(os.path.join(dirpath, filename),
                                      os.path.join(fgdbFolder, "output.gdb", outFilename))

Re: Uploading and using data to a geoprocessing service

meriyalootka — Sun, 12 Dec 2021 03:20:54 GMT

import arcpy
import zipfile
import os

inFile = arcpy.GetParameterAsText(0) 

# Create a folder in the scratch directory to extract zip to
zipFolder = os.path.join(arcpy.env.scratchFolder, "zipContents")
os.mkdir(zipFolder)


# Extract the zip contents
zip2Extract = zipfile.ZipFile(inFile, 'r')
zip2Extract.extractall(zipFolder)
zip2Extract.close()

# Create a folder in the scratch directory to hold the fgdb which will be downloaded
fgdbFolder = os.path.join(arcpy.env.scratchFolder, "fgdbOutput")
os.mkdir(fgdbFolder)

# Work through all the FeatureClasses inside the extracted zip folder
for dirpath, dirnames, filenames in arcpy.da.Walk(zipFolder, datatype="FeatureClass"):
    
    for filename in filenames:
        
        # You could replace the code below here with your own code to do what you want....
        arcpy.AddMessage("Copying: {}".format(filename))
        
        # Strip .shp from the filename when merging shapefiles
        if filename.endswith("shp"):
            outFilename = filename[:-4]
        else:
            outFilename = filename

        # Copy each featureclass into the output.gdb
        arcpy.CopyFeatures_management(os.path.join(dirpath, filename),
                                      os.path.join(fgdbFolder, "output.gdb", outFilename))

Hy
Many thanks
Please help me about input and output parameters.

Re: Uploading and using data to a geoprocessing service

meriyalootka — Sun, 12 Dec 2021 03:20:56 GMT

import arcpy
import zipfile
import os

inFile = arcpy.GetParameterAsText(0) 

# Create a folder in the scratch directory to extract zip to
zipFolder = os.path.join(arcpy.env.scratchFolder, "zipContents")
os.mkdir(zipFolder)


# Extract the zip contents
zip2Extract = zipfile.ZipFile(inFile, 'r')
zip2Extract.extractall(zipFolder)
zip2Extract.close()

# Create a folder in the scratch directory to hold the fgdb which will be downloaded
fgdbFolder = os.path.join(arcpy.env.scratchFolder, "fgdbOutput")
os.mkdir(fgdbFolder)

# Work through all the FeatureClasses inside the extracted zip folder
for dirpath, dirnames, filenames in arcpy.da.Walk(zipFolder, datatype="FeatureClass"):
    
    for filename in filenames:
        
        # You could replace the code below here with your own code to do what you want....
        arcpy.AddMessage("Copying: {}".format(filename))
        
        # Strip .shp from the filename when merging shapefiles
        if filename.endswith("shp"):
            outFilename = filename[:-4]
        else:
            outFilename = filename

        # Copy each featureclass into the output.gdb
        arcpy.CopyFeatures_management(os.path.join(dirpath, filename),
                                      os.path.join(fgdbFolder, "output.gdb", outFilename))

I need a model for extracting a zip file (contain shape file) to a feature class. I need to use from extracted shape file in my model. please help me more.

Re: Uploading and using data to a geoprocessing service

meriyalootka — Fri, 25 Jan 2013 15:41:02 GMT

I need a model for extracting a zip file (contain shape file) to a feature class. I need to use from extracted shape file in my model. please help me more.

Re: Uploading and using data to a geoprocessing service

KevinHibma — Fri, 25 Jan 2013 15:41:02 GMT

You have to use a script tool. Theres no tool in the toolbox (to put into modelbuilder) to extract a zipfile.
You can wrap the above script as a script tool and put that into a model if it serves your purpose.

Re: Uploading and using data to a geoprocessing service

meriyalootka — Fri, 25 Jan 2013 15:43:18 GMT

You have to use a script tool. Theres no tool in the toolbox (to put into modelbuilder) to extract a zipfile.
You can wrap the above script as a script tool and put that into a model if it serves your purpose.

I,m trying more than 1 month, but result is very bad. I tested last script, but can not use it!!
Last script return this error:
IndentationError: unexpected indent YourScript.py, line 1)
Failed to execute (Your script)

please give me a model if you can.
This is my email:meriyaloot@gmail.com

Regards

Re: Uploading and using data to a geoprocessing service

meriyalootka — Sun, 27 Jan 2013 15:12:51 GMT

You have to use a script tool. Theres no tool in the toolbox (to put into modelbuilder) to extract a zipfile.
You can wrap the above script as a script tool and put that into a model if it serves your purpose.

Hi esri
I,m a bad luck user. There isn't no one to solve my problem:mad:

Re: Uploading and using data to a geoprocessing service

meriyalootka — Wed, 30 Jan 2013 18:13:10 GMT

You have to use a script tool. Theres no tool in the toolbox (to put into modelbuilder) to extract a zipfile.
You can wrap the above script as a script tool and put that into a model if it serves your purpose.

Hi
The script return this error:

Traceback (most recent call last):
File "C:\Users\administrator\Documents\importzip.py", line 22, in <module>
for dirpath, dirnames, filenames in arcpy.da.Walk(zipFolder, datatype="FeatureClass"):
AttributeError: 'module' object has no attribute 'Walk'

What is wrong in my setting?

Re: Uploading and using data to a geoprocessing service

KevinHibma — Thu, 31 Jan 2013 14:32:58 GMT

My apologies, the da.Walk command was added at 10.1 Service Pack 1. Based on the message I'm guessing you have 10.1 final (no service packs). If you're able to install the service pack it'll fix that.

Re: Uploading and using data to a geoprocessing service

meriyalootka — Sun, 03 Feb 2013 16:54:09 GMT

My apologies, the da.Walk command was added at 10.1 Service Pack 1. Based on the message I'm guessing you have 10.1 final (no service packs). If you're able to install the service pack it'll fix that.

Hi
Esri published his Unzip.py about 8 years ago. Why I should use a script that just can work with AGS 10.1 SP1?
I have a Server that has AGS Server 10, and another server has AGS Server10.1, I need a solution that can work with AGS Server 10. please help me more.
Regards