topic Re: Publishing a geoprocessing toolBOX in ArcGIS Enterprise Questions

Publishing a geoprocessing toolBOX

KevinGooss — Tue, 03 Jul 2012 13:37:59 GMT

I see how i can publish a gp tool by creating the tool, running it in arcmap, and then selecting the results and clicking Share As...
That creates an sd file that i can then use in ags manager to publish my tool.
The problem is that i have a toolbox with 30 tools in it. I don't want to run all 30 and go through this process. Can't i just publish the entire toolbox somehow?

Also, why the change in publishing gp services? I liked being able to go into catalog, connect to my ags, then right click and add new service based on a python scripted tool.
that seemed like a great workflow because i could then edit the python script in the background without having to reup my service (unless i changed params).
It appears that under the new workflow I cannot modify my python script without having to open arcmap and rerun everything.
This will add much time to my dev process.

Re: Publishing a geoprocessing toolBOX

KevinHibma — Tue, 03 Jul 2012 16:14:52 GMT

The idea behind publishing a good result, is that because it ran successfully in ArcMap, it has much higher odds of being successful as a service.
It makes more sense to do the dev/testing inside ArcMap with a local tool than it does to publish a service and modify that.
Additionally by publishing a result, we're able to help enforce best practices and modify tools/scripts/models that otherwise wouldn't have published successfully. Once something is a service and you modify it, we have no way to help with best practices or warn if something isn't supported. When you do all your dev work up front and publish the result (hopefully you only need to publish once), we're able to tell you what might be wrong with it and suggestion action.

Well I dont really suggest this, if you're only making minor edits to your script, you can modify the script which gets moved to the Server (understanding that these edits dont persist back to your original script).
You'd find the script powering your GP Service at:
C:\arcgisserver\directories\arcgissystem\arcgisinput\<service name>.GPServer\extracted\v101\<original folder that held toolbox name>

Re: Publishing a geoprocessing toolBOX

KevinGooss — Tue, 03 Jul 2012 17:21:49 GMT

Thank you for the prompt answer Kevin. I have to say that is a little disappointing. As a developer I expect 'best practices' to be a document that i can digest and choose to follow - not a directive from the software telling me what i have to do.

The architecture you describe creates a deployment nightmare for me. Now i am required to completely replicate the target environment in order to publish what you consider to be a 'good' tool. There are perfectly valid scenarios were it makes more sense to publish a service and modify that on the target environment. It appears that option has been taken away from us.

For one project alone I have 30 custom python Script Tools. You are telling me that for a single deploy I need to run each of those in ArcMap and then publish the results or add them as tasks to an already existing service and then overwrite that. And if i have a fix for just one of those scripts I am now expected to edit that locally, retest all 30 in ArcMap and then overwrite the published service? And if that is a remote environment without ArcMap I have several more steps to create the sd file move it to the remote environment and then republish there?

So a single bug fix that used to require only an edit to a py file now requires much, much more. Additionally, this republish is going to kick out every user who happens to be using any one of those 30 tasks at the time?

I think perhaps that I am just confused. I'm sure there is a simple explanation and I'm just overlooking it. Could you correct me and point out the simple path to custom tool deployment that doesn't involve ArcMap and Results?

Re: Publishing a geoprocessing toolBOX

KevinHibma — Tue, 03 Jul 2012 18:23:18 GMT

Please let me clarify some statements:
When I say best practices, I mean best practices being suggestions, and the addition of trapping for the things which prevent your service from working.
Best practices appear as warnings and messages in the analyzer and others are the errors which wont allow you to publish the service.
Warnings and messages are something you (the service publisher) can chose to ignore or implement. We don't force these on you.

I understand your concern with re-running and adding 30 tool executions into a task. Honestly, throughout beta nobody came to us with more than 3 or 4 tasks per service presenting a concern regarding running this many tools. To me, 30 seems like a lot of tasks on a single machine, but if you're deploying that many then you must have the system resources to deal with that many running instances.

At this point all I can do is offer you some suggestions to reduce the time it takes to republish 30 tasks. Hopefully you can make use of one of these in your publishing:

1) In the Geoprocessing Options in ArcMap (Geoprocessing > Options), you can set Results Management to "Never Delete".
With this setting, run every tool you want to publish, you'll get a result for each
Save the MXD.
Anytime you want to republish the service, you simply need to open that MXD, run the updated tool, and then put together the service with a combination of old/new results.
>This method is also handy as if any data required for the service has gone missing, the analyzers should catch it.

2) Go to Windows > ArcMap options > Sharing Tab > check "show file location when saving draft service definitions"
Run all of the tools you want to publish (you'll get a result for each)
Get into the Service Editor by sharing a result as a service
Add all your Results into the service
Click the "x" on the service editor, and say YES to saving a draft.
Note where the draft file gets saved to.
You can re-open these and add or remove results from it later (when you need to update).
Note: The .SDDRAFT files act more like pointers, so if you start moving tools/data from where they were when you originally saved, you could break references.

3) For 10.1 sp1 we're working on a pythonic (arcpy) way to publish gp services.
It is early in the testing and looks good so far, but till its 100% I can't guarantee it'll make sp1.
In short, you could write a runner or execution script to power you through the publishing process.
The script calls each toolbox/task, runs it, assigning each to its own result object.
You take each result object and pass it into arcpy.GPCreateSDDraft([list of result objects], , , ) - new function, then Analyze it, Stage it into a .SD, and finally Upload.
This entire workflow has been well documented and exists for Maps. Like I said, we're working on it for 10.1 sp1 in terms of GP. Hopefully it'll come together like we want.

Re: Publishing a geoprocessing toolBOX

KevinGooss — Thu, 05 Jul 2012 16:53:40 GMT

Thank you for the quick reply Kevin.
It looks like this may not be so bad after all. Just noticed your ArcGIS Server Administration Toolkit, that should help as well.
We do maintain a significant number of geoprocessing tasks and i think running them all in arcmap is going to be the toughest part.
but it seems like esri is moving in the right direction in terms of allowing python programmers greater control of the processes involved in make gp services.
And i also like that fact that the more 'wizard'-like tools (like publishing from arcmap) seem to use scripting and services at their heart.
thanks again for the suggestions.

Re: Publishing a geoprocessing toolBOX

AaronBarkhurst — Wed, 22 Aug 2012 14:01:59 GMT

I have to agree with KG22. This new publishing method has made deployment of our internal system a NIGHTMARE!!! This process has now introduced SO many more steps to what used to be a relatively seamless and simple process because of the number of scripts contained in my toolbox. I also have a number of scripts in a toolbox that are dependent on outputs from other scripts or json inputs from our front-end that make running each one before publishing extremely painful, and frankly, a waste of time! By doing this, ESRI has single handidly increased my workload and time it takes to do my job. I have also noticed that when publishing my scripts, ESRI takes it upon themselves to REWRITE my code in the service directory, which I have found actually causes some of my script to FAIL as GP services....WHAT!!!!!!! That is a HUGE no no to me. DO NOT CHANGE MY CODE!!!!!!!!!!!!!!!!!!! Please fix this in SP1, as I am extremely disappointed with the change!

Re: Publishing a geoprocessing toolBOX

KevinGooss — Wed, 22 Aug 2012 14:12:24 GMT

Hopefully this will be addressed in sp1 with a way to deploy scripts directly from python.
The idea of esri "running" my script and determining if it will work and is optimized is anathema to the development process.
I have over 10 years experience in jumping through hoops to get software to work - that is being challenged now.
I guess one way around this (that we are kicking around now) is to publish a (basically) blank python script the way esri suggests (via map results) and then hunt down where ags has copied and hidden the script and go there and edit the script to be what you want.
That strategy is probably not going to work well if you have to modify input/output parameters.

Re: Publishing a geoprocessing toolBOX

AaronBarkhurst — Wed, 22 Aug 2012 14:16:39 GMT

KG22,

That is funny that you mentioned the workaround of publishing a blank script and then editing the file in the service directory. That is exactly what we have done as well. We just create a script that does nothing but import the arcpy module and then publish that, and edit the python file with the proper code.

Re: Publishing a geoprocessing toolBOX

KevinGooss — Wed, 22 Aug 2012 14:21:39 GMT

I think we are going to go the route of converting to the new native python toolboxes rather than the older 10.0 way of py scripts tied to tools. It seems esri is going in that direction so might as well get on board. if we can publish them directly from python that will take desktop out of the picture and be very nice.
I can understand wanting the script to be runable because it makes debugging a little easier because you are in arcmap. But there are scripts that are just too complex to be setup in arcmap to run every time you want to make a small change.

Re: Publishing a geoprocessing toolBOX

KevinHibma — Wed, 22 Aug 2012 14:32:27 GMT

I have also noticed that when publishing my scripts, ESRI takes it upon themselves to REWRITE my code in the service directory, which I have found actually causes some of my script to FAIL as GP services....WHAT!!!!!!! That is a HUGE no no to me. DO NOT CHANGE MY CODE!!!!!!!!!!!!!!!!!!! Please fix this in SP1, as I am extremely disappointed with the change!

The only code change we make to scripts is the insertion of variables for data (and sometimes sql expressions). Since part of the new publishing processes is a mechanism to ensure all data required by the service can be found and is available, we need to update the paths.
If you're encountering an instance where we didn't update the paths to data correctly, or didnt copy data over that was required then I'd be very interested in fixing it. Is there any chance you can share the tool/script/some data with me?

Unfortunately based on timing, we're getting SP1 pretty locked down. I'd need to be able to reproduce the issue here in house and get a fix for it before Friday to have any chance at SP1 - else it would probably be next service pack.
If you can share, feel free to email me directly at khibma@esri.com

Re: Publishing a geoprocessing toolBOX

MattMoyles — Wed, 22 Aug 2012 14:43:24 GMT

I've got to agree with Kevin Gross, and Aaron Barkhurt. I hope some ESRI employees read this thread because you guys really seem to have dropped the ball on 10.1 when it comes to publishing.

My first complaint with publishing is that you cannot publish an arcobjects executable as a GPService on an ArcServer. This is a major regression. I don't believe it's your place to say if it's a security vulnerability or not. That's up to our team. I wasted 2 expensive support tickets to find out that you can't publish an executable. I ended up having to wrap it in a python script calling my executable as a sub-process. It feels hacky to say the least.

I also agree with everything that Aaron Barkhurst said about publishing python GP scripts. Changing my code at publish time is an insult to the developer. You guys seem to be catering to a crowd who is not interested in learning arcpy or programming in general at the expense of power users.

Model Builder is a useless tool. It may look good on your powerpoint slides at the ESRI conference but it promotes not learning how to program. Organizations that use model builder can be found using tools like Entity Framework and DevExpress. Your code is likely very weakly tested and buggy. Enforcing the publishing successful of results is just catering to this crowd, again, at the expense of power users.

We have a ton of python classes and libraries and encapsulating them into their own working directory at publish time just does not work, especially when publishing to the ArcServer is rewriting my code and changing paths and directories. We ended up doing the same thing listed above. Which is to just publish a dummy python script and then going into the ArcServer system folders and copying in the real script.

We are now tasked with trying to implement a continuous integration pipeline from source control to published gp services on our development ArcServer. If it's able to be done it's going to be very hacky.

So I will end by saying this. ESRI should be giving developers tools that empower them and make their lives easier, and not boxing them in because some people are too lazy to learn.

10.1 has got me interested in looking into PostGIS.

Re: Publishing a geoprocessing toolBOX

KeG — Wed, 29 May 2013 12:44:51 GMT

I am in the same boat here. Horrible experience just trying to publish a simple toolbox with three tools in it. Analysis takes 30 minutes. Publish takes an hour. And that is for ANY change i make in ANY script in those 3 tools.
How is this possibly a working solution?
I could create separate services for each tool but that is overkill. These tools are related and should go in a toolbox.
Sp1 didn't improve upon anything. The script solution adds many, many more steps into the process. I have to save out results, create an sddraft, turn that into and sd, take that and then publish it? Come on.

Re: Publishing a geoprocessing toolBOX

NikolasStevenson-Molnar — Tue, 25 Jun 2013 22:25:39 GMT

I'd like to add by 0.02 and provide a workaround for those struggling with this problem.

First, I agree with many of the previous posters that the process for deploying GP tools to server is extremely kludgy and unnecessarily burdensome. It's un-intuitive (publish from the result of a tool run?), and makes a number of problematic assumptions: namely that you're deploying from a machine with ArcGIS Desktop, that you're deploying individual tools, not toolboxes (unless I've missed an option?), and that the tools are even intended to run in a Desktop environment. Additionally, the extra burden to deploy tools does very little to ensure tools are fully functional before deployment. Any tool with even a bit of complexity will have multiple execution paths and what may succeed with one set of inputs may fail with another. The only way to ensure the functionality of all execution paths is to write good tests, and frankly you can't force people to do that; they either will or they won't.

Now, for a workaround. You can by-pass the whole run->puslish-from-result process by using the ArcGIS Server REST API. If your sources are already present on the target server (e.g., my deployment process involves pulling sources from a Mercurial repository onto the target server), then you can POST to the "createService" admin endpoint with JSON-formatted service info to create the service (http://resources.arcgis.com/en/help/server-admin-api/createService.html) and then POST to the "startService" admin endpoint (http://resources.arcgis.com/en/help/server-admin-api/startService.html) to start the previously created service. Wrap this all up into a deploy script and you'll be set.

If your sources aren't already on the server, I imagine you could use the upload API to get them to the server, though I haven't tried this approach.

Server is sort of finicky about creating and starting services, so be prepared for a bit of trial and error. Often, if you get something wrong, it will let you create the service, but you won't be able to start the service. If your service fails to start, check the server log for clues about why. I found that the "outputDir" and "virtualOutputDirectories" parameters are actually required despite being reported as "optional" by the API docs. If you create your service without those parameters, then it will fail to start.

Cheers, and happy deployments!
_Nik

Re: Publishing a geoprocessing toolBOX

DaveHighness — Wed, 26 Jun 2013 14:58:57 GMT

I agree with these posts. Publishing Python GP Tasks has become a big pain under 10.1. I now routinely create two scripts, the real one and a dummy one with just the input and output parameters. I publish the dummy and replace it with the real one once published. This saves me some time in publishing but I still can't publish a toolbox at once like I did in 10.0. It shouldn't have to work that way. Why can't I be trusted to test and debug my own scripts? Please give those of us who know what we are doing to ability to force publish a toolbox as is!

Thanks, Dave

Re: Publishing a geoprocessing toolBOX

KeG — Wed, 26 Jun 2013 15:17:36 GMT

Couldn't agree more with the last two posts.
You made some great points _Nik and thank you very, very much for those tips.
between your method and Dave's that's two new ways around this issue.

I have resorted to going back to the old tbx method and am not using the pyt python toolbox that i was so looking forward to.
I could never get the calls out to resources to work the way i wanted.
I have 45 gp tools that i need to publish and the pyt file would be enormous if i used it as anything other than just a placeholder for params and a pointer to py files.

To publish those 45 tools took me over 5 days! That is cost to me and to my clients. Now i have a (precious) mxd with successful results for every one of them. However, i had to publish them all as separate services because i couldn't just publish a toolbox. This is clutter in my services directory.

ESRI needs to undo this in my opinion. I think they were going for a level of handholding that will allow casual desktop users to start cranking out services. But that is not the way i use the software.
And there is a marked difference between a gp script that is designed for desktop and one designed for ags publishing. they need to recognize that and split the workflows - not try to incorporate them.

Re: Publishing a geoprocessing toolBOX

LakshmananVenkatesan — Thu, 13 Mar 2014 22:01:22 GMT

All,

I am scared of reading all the threads. We have developed custom geoprocessing tools using ArcObjects by extending
Geoprocessing framework. We have 70+ tools with wide variety of inputs. This tool box has been published as
Geoprocessing service in ArcGIS Server 10.0. This was simple and straightforward exercise.

But I dont think this can be done easily in ArcGIS Server 10.2 version. I am still figuring out the way to achieve this. if any one has some experience please point me on this. Our plan to migrate to 10.2 is mainly depend on this. I cannot run a each tool and generate a result and share them as service is NOt possible, because every tool input might be different based on work flow. I dont see this as a alternative.

Please throw some light on this.

Re: Publishing a geoprocessing toolBOX

AndreasRuloffs1 — Wed, 02 Apr 2014 12:47:36 GMT

Hi,
I don't like the new deployment either.
Our normal workflow is that we develop the tools in our system and later publish the well ready developed and well tested at our customer. Especially when deploying at the customer a too complicated and too long workflow does not make us look good.

But I have another ugly thing with publishing. Our Scripts normally contain several files, the script itself, a config script and a general helper script. The last ones are imported in the first one.
Publishing the script copies only the script itself but not the other files, so after publishing executing the script fails.
I have to manually copy the files into the path Kevin mentioned earlier.
I have registered the path where the scripts came from as data store, but this does not prevent the copy.

Bye,
Andreas

Re: Publishing a geoprocessing toolBOX

DanielGarcia — Tue, 02 Sep 2014 10:37:24 GMT

I agree with these post, but I think that Nikolas give us the best option. Actually I use the "createService" rest service method on the root or desire ArcGIS Server folder. Like this...

http://servername:6080/arcgis/admin/services/foldername/createService

I use the attach json code to publish a tbx file. This tbx has a lot of tools that reference to many others python scripts. All of this are located into a registered folder to ensure that ArcGIS Server has permissions.

This process give me the best way to publish tbx.

Regards

Dani