topic Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script in Geoprocessing Questions

10.1 ExtractData.py script not the same as embedded Extract Data tool script

GerardoGarza — Tue, 07 Aug 2012 15:45:47 GMT

I have been creating services in 10.1 using geoprocessing tools created in 10.0.
One tool was based on the Extract Data Task in the Server Tools toolbox.
I ran the tool which completed successfully, but when I tried to publish results as geoprocessing service, I received the following:

Error - Code 68 - Script Extract Data contains broken project data source: gdb
Warning - Code 24032 - Data source used by Script Extract Data is not registered with the server and will be copied to the server: gdb

I then used the 10.1 ExtractData.py script to refactor the 10.0 script and received the same error.

When I ran the Extract Data Task directly from the Server Tools toolbox, it published successfully.
When I copied the Extract Data Task and script to a new toolbox, publishing failed again with the above error.
After some more research and testing, I realized that the Server Tools Extract Data was using an embedded script which was different from the Extract Data.py script in the ArcGIS\Desktop10.1\ArcToolbox\Scripts folder.

Question: Is it possible to get copy of the functional embedded ExtractData.py script?

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

SteffenSchillinger — Sun, 09 Sep 2012 18:48:52 GMT

Hi, I'm having the same problem and contacted ESRI support about it because I think it's a bug. Could you solve it or has anyone sent you some hints? Why do you think the tools run embedded scripts that are different from those in ArcGIS\Desktop10.1\ArcToolbox\Scripts folder? I tested this with the ExtractData.py script and added a message command to the original code and it shows up when I execute the original Server Toolbox model.

Best,

Steffen

steffen.schillinger@gmx.net

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

KevinHibma — Mon, 10 Sep 2012 14:26:04 GMT

This shouldn't have anything to do with the ExtractData script, nor matter if its embedded or not.
The analyzer error 00068 says that the packaging/publishing process cannot find required data to complete publishing. (broken project data required for the service)

I've never seen that error pop up in regards to the Extract Data task.
We'll bang on it here to see if we can reproduce it, but I'm sort of at a loss as why thats happening for you.
Instead of publishing as a service, can you save it as a geoprocessing package - same error, or no?
Can you very clearly spell out your workflow? (maybe you're doing something I'm not...?)

thanks

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

SteffenSchillinger — Mon, 10 Sep 2012 15:24:14 GMT

Dear Kevin, thanks for answering. The error also pops up when trying to publish a package. The workflow follows the example at http://resources.arcgis.com/en/help/main/10.1/index.html#/Clip_and_ship/005700000073000000:

1. Copy Extract Data Task from System Toolboxes\Server Tools\Data Extraction to MyToolboxes\New Toolbox
2. Edit model and set layers to clip and other parameters as in example
3. Execute Model -> works
4. Publish Model as Geoprocessing service -> works, no errors when analyzing (underlying data of MXD have already been registered with the server and been published as map service)
5. Copy Extract Data Script tool from Server Tools\Data Extraction to MyToolboxes\New Toolbox
6. Copy ExtractData.py from ArcGIS\Desktop10.1\ArcToolbox\Scripts to different folder on local drive (i.e. c:\work)
7. Set source property of MyToolboxes\New Toolbox\Extract Data model script to c:\work\ExtractData.py
8. Delete Extract Data model tool from MyToolboxes\New Toolbox\Extract Data Task model
9. Drag and drop MyToolboxes\New Toolbox\Extract Data to model and reassign/define parameters and links
10. Execute model -> works
11 Publish model as geoproc service -> error pops up when analyzing

Resume: same model script without any modifications causes this error when replacing original script tool with it in Extract Data Task model

regards and thanks,

Steffen

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

KevinHibma — Mon, 10 Sep 2012 16:05:46 GMT

Steffen,

Thanks for the detailed steps.
Using steps 6-11, on a ArcGIS 10.1 final machine I can reproduce the issue.
Using a 10.1 SP1 machine (we havent released SP1, we're finalizing it right now), I cannot reproduce the issue - so we've fixed it.
My apologies for the inconvenience.

A question though - why do you need to copy the script and model tool to a custom toolbox?
If you want to control parameters (for example, you want to remove options from the feature choice list, or you dont want to expose the raster format parameter) you can run the tool from the system toolbox, and then during publishing modify those parameters inside the service editor. Just click the parameter name on the left hand side and change the Input Mode, or configure the Choice List options. Unless you're adding new values to the data extract format task, everything else you want to do should be configurable through the service editor.

I hope this will provide a suitable workaround till SP1.

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

SteffenSchillinger — Mon, 10 Sep 2012 17:52:11 GMT

Dear Kevin, thanks a lot for checking this so quickly. I was afraid I had to wait for SP1 - unfortunately, I have another incident pending (Silverlight Viewer can't display image service as basemap) and support told me that SP1, which would be released later this month, will fix this, too. Isn't there a way through EDN or so to get a beta version of SP1? My task requires to customize the Python source code of ExtractData.py and I thought best way to start is using a copy of the original file. Do you think I could publish a geoproc service building the model and model script from scratch?

In any case, many thanks for clarifying this. You were faster than ESRI support 😉

Cheers,

Steffen.

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

KevinHibma — Mon, 10 Sep 2012 17:58:07 GMT

No, we dont do "beta" service packs, only the final release of them.

If you absolutely require a custom version of the extract data task, do you still need feature parameter input? This is the problem, if you can either hardcode or change that parameter, you can probably get around the analyzer problem.

The tool does a name match, so its possible (i havent tried this though) to play with the name inside the parameter and then change the code in the tool. Of course if you dont need the feature type parameter, the best solution would be to hardcode it in the script and not expose that parameter. Again, I think this will work, but no guarantees.

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

SteffenSchillinger — Mon, 10 Sep 2012 18:23:08 GMT

Hi again,

you mean the Value List in the Feature Format parameter causes the problem? Because it uses the string ".gdb"? That's why it's looking for a data source named "gdb"? Well, thanks for the hint! I can hardcode-limit this to .shp only until SP1 is available. I'll give it a try and post my results.

Steffen.

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

KevinHibma — Mon, 10 Sep 2012 18:46:02 GMT

Hi again,

you mean the Value List in the Feature Format parameter causes the problem? Because it uses the string ".gdb"? That's why it's looking for a data source named "gdb"? Well, thanks for the hint! I can hardcode-limit this to .shp only until SP1 is available. I'll give it a try and post my results.

Steffen.

exactyl! 🙂

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

SteffenSchillinger — Mon, 10 Sep 2012 19:24:56 GMT

No, we dont do "beta" service packs, only the final release of them.

If you absolutely require a custom version of the extract data task, do you still need feature parameter input? This is the problem, if you can either hardcode or change that parameter, you can probably get around the analyzer problem.

The tool does a name match, so its possible (i havent tried this though) to play with the name inside the parameter and then change the code in the tool. Of course if you dont need the feature type parameter, the best solution would be to hardcode it in the script and not expose that parameter. Again, I think this will work, but no guarantees.

Hi Kevin, I tried this but without success. I even completely deleted the two format type variables from the model and hard-coded the getparameter string to shapefile. The model runs and does its job but the analyzer error remains.

Besides, why would this error occur only when the script is not in it's original location? Seems a bit odd to me. SP1 addresses this error for example

NIM081276 - Sharing some script tools that use Network Analyst tools as a geoprocessing service incorrectly raises analyzer error "Results contains
broken project data source".

, and since this seems to fix the before mentioned error, too, I guess there are things wrong in the model builder itself.

Best,

Steffen

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

GerardoGarza — Wed, 19 Sep 2012 21:56:48 GMT

Hi Steffen,
Sorry for the delay in returning to this thread. I was pulled to another project that took my attention away.

The issue lies in the strict scanning of python scripts when publishing as a geoprocessing service. According to Authoring Geoprocessing Tasks with Python Scripts, "when your script is scanned, every quoted string (either single- or double-quotes) used in a Python variable or as an argument to a function is tested to see if it is a path to data that exists."

Line 135 in ExtractData.py is the culprit. I changed "gdb" to "file geodatabase" so it wouldn't look like part of a path when passed to getTempLocationPath, and it published. I'll be curious to see how the script has changed in SP1.
Hope this helps,
GG

Re: 10.1 ExtractData.py script not the same as embedded Extract Data tool script

TysonSchoepflin — Mon, 10 Feb 2014 16:20:03 GMT

Hi Steffen,
Sorry for the delay in returning to this thread. I was pulled to another project that took my attention away.

The issue lies in the strict scanning of python scripts when publishing as a geoprocessing service. According to Authoring Geoprocessing Tasks with Python Scripts, "when your script is scanned, every quoted string (either single- or double-quotes) used in a Python variable or as an argument to a function is tested to see if it is a path to data that exists."

Line 135 in ExtractData.py is the culprit. I changed "gdb" to "file geodatabase" so it wouldn't look like part of a path when passed to getTempLocationPath, and it published. I'll be curious to see how the script has changed in SP1.
Hope this helps,
GG

Ran into a similar issue.

ArcGIS Server: 10.11

Example source: Geoprocessing service example data provided with ArcGIS for Desktop version 10.1 and 10.2. http://www.arcgis.com/home/item.html?id=6987d0a0daa14fe9861bae93e4808845

Example: ClipAndShip > ExtractZion > ExtractZionData

Trying to share as a geoprocessing service throws the same error and warning. Changing line 135 in the python script to "file geodatabase" does fix the error. However, layer groups are still a problem. Any layer group in the file is given a warning code 24032. Along with the layer groups is another 24032 warning for "\" as shown here: "Data source used by Script Extract Data is not registered with the server and will be copied to the server: \".

A "flat" map file with no groups is a way to get around "copying" data to the server. If you publish with the "\" warning, the selection box is empty and you can publish anyway.

My question is why are layer groups and "\" treated as data not on the server? Is it required to use "flat" (no layer groups) map files to publish geoprocessing services?