Best practices for publishing GP Services from Python?

[davedoesgis]

I'm pretty experienced with Python for geoprocessing (arcpy) and Enterprise/Online (ArcGIS API for Python), but new to creating GP services. I like to code Python in my IDE for debugging, code hints, etc. As I get started with geoprocessing services, I'm spending my time in Pro and the Idle editor, which is not where I want to develop complex code. Then, we have to share this to a service from our GP history, rather than the actual source code, which seems odd. Esri's docs say we can develop our code by shuttling it back and forth to our IDE via copy/paste, which seems inefficient. 

I have some specific questions below, but mostly I'm just looking for best practices and workflows. What's the best way to go from a full dev experience in a good Python IDE (Eclipse/PyDev, VS Code, PyCharm, etc.) to a  working GP Service? Tips and tricks, please!!!

Deployment: Esri has a Python sample to publish a GP service. It's not even using the latest .atbx format and publishes to AGS (not Portal), so it seems quite outdated, but might work. Is something like this how to deploy with Python? 

Toolbox format: There's the .tbx, .atbx, and .pyt formats. The .pyt format seems the most straightforward to edit, but I just get a 999999 error trying to share to Enterprise (version 10.81). What format do you use? 

Code structure: One thing I thought was to make my GP service just a basic shell and then deploy my main application logic separately. I should be able to add my own modules to the PYTHONPATH. I was even able to just drop Python modules into the same folder as my .atbx file on ArcGIS Server and it imported them ok.

Anyhow, these are the ideas bouncing around in my head, and I'm sure there are other issues I haven't run into yet... I'd like to hear from others how they take complex Python code from their IDE to a GP service. Thanks!

[mody_buchbinder]

I done a few gp services, here is my ideas.

I use python editor (like pycharm) and create gp tool that pointing to my external python file. This way you can use any editor.

You cannot use the debugger of the editor but you can do it if you replace the arcpy.GetParameter with hard coded values.

About deployment why not using the desktop software without code. You are not doing it too many times.

In Pro you right click in the history and use share.

 The publish process change your scripts so it can run as a service.

You can understand the process better if after the publish you go to C:\arcgisserver\directories\arcgissystem\arcgisinput\MyTool.GPServer\extracted\p20 and see the updated script. You can do changes on this script just for testing (this is not supported) but really to understand how to build you script so it will converted correctly.

You can set the message level so you will see all the arcpy.AddMessage on the client.

It is not always easy but you can make any script as a service - it is very strong 

Have fun

[ZacharyHart]

You can do changes on this script just for testing (this is not supported)....

Not supported but often required, at least in my experience. The publishing operation often butchers data source references. Background: I've not found anyway to create a GP service which can reference feature layers (even if your local script tool happily consumes and appends data) so you're stuck with direct DB connections (referencing SDE file connections). Link to post here 

[ZacharyHart]

Replying for visibility as I'm also interested in this.

I've had more success with publishing GP services from tools built with Model Builder, but MB always leaves me wanting and I usually end up building in Python.

@davedoesgisI'm still using mostly tbx and haven't made the leap to pyt. Also, if you're going to be pulling in your own custom functions into a script, you're going to have to get those up to your server environment. If they're simple, you may just want to include those functions within the code of your script tool instead. (Lame cop-out I know).

I'm currently chasing an issue while trying to rebuild a tool from model builder with python. Works flawlessly locally, and then all the 'fun' begins when trying to publish. My previous model leveraged the %scratchWorkspace% and %scratchGDB% environment shortcuts and that has always been bullet-proof for me when publishing. But this isn't working out so well with the script tool. I'm using a feature set for interactive selection and that polygon gets written to the default workspace as defined in your environment settings in Pro. Problem is, when publishing, AGS says 'Hey, i don't know where c:\users\userstuff\folders\default.gdb resides so I'm going to have to copy that data over'. 

Similar to what @mody_buchbinder  offered: message the heck outta everything it will help with debugging.

[mody_buchbinder]

I have the problem with changing the path too.

The solution I found:

db_connect = r"\\networkDisk\file.sde"

 tab1 = os.path.join(db_connect , r"database_table")

The \\networkDisk must be registered in the server as directory that the server can access.

This syntax works for me and the publish did not change anything.

[ZacharyHart]

Thanks. Similar to what was suggested in this thread, but unfortunately still didn't have luck with that.