Publishing PYT with a string pointing to an SDE file on disk crashes ArcCatalog

[Anonymous User]

Hello all,

I have encountered a new problem. Consider the code snippet below. This code can be dropped into any Python Toolbox (*.pyt) in place of the execute() function. All it is doing is assigning a plain text string that contains a file path to an sde file on disk. The same value is assigned on two different lines. In the first case, the value is all in one string, while in the second case, it is broken apart.

Either line works fine when running the tool in ArcGIS desktop. However, if I attempt to publish the result of the tool to ArcGIS server, ArcCatalog crashes completely, with presenting the dialog for reporting a problem. If that first line is excluded, and/or if the path does not match the location of an existing sde file, then publishing the will work fine.

 def execute(self, parameters, messages):
  
  ### Declaring a text string that points to an existing *.sde file on the local
  ### disk will cause ArcCatalog to crash when attempting to publish
  ### a result from running this tool as a GP Service
  myGeodatabase = "C:/Data/test.sde"
  
  
  ### The following approach seems to
  ### workaround this for now:
  myGeodatabase = "C:/Data/"+"test.sde"
  
  return

I'm guessing this may have something to do with the way that ArcGIS 10.1 keeps track of references to data on disk, and aligns these with databases/folders that are registered on a ArcGIS server that I'm publishing to. If that's the case, then I would imagine that the workaround I show is tricking the publishing service into overlooking the reference to the sde file. However, even if I have both the database connection and the folder that contains the sde file registered as data sources on the server, the publishing service will still crash.

Has anyone else encountered this kind of problem, and/or is there something particularly wrong that I'm doing with this? Perhaps there is a better method to handle references to local data - would anyone have suggestions on that? It doesn't make sense in my case to have the location of the sde file exposed as a parameter in the GP tool, as client applications will not be aware of this...nor would I want clients to supply the location of a different sde file on the server.

[Anonymous User]

Ok - I've done some further testing...I can avoid adding the string to the Python code itself, by adding a Workspace parameter to my tool in the PYT toolbox. When this gets published, the Workspace parameter is treated as a 'constant' value, and doesn't get exposed to clients through the REST interface.

However, I still have a problem with the published GP tool getting file references rewritten to the wrong location. For example, here is my parameter as it is defined in the PYT toolbox:

  TestWorkspace= arcpy.Parameter(
   displayName="TestWorkspace",
   name="TestWorkspace",
   datatype="Workspace",
   parameterType="Required",
   direction="Input")

  ### Cannot set a default value in code - this will
  ### crash ArcCatalog when the tool result is published:
  # TestWorkspace.value = "C:/Data/test.sde"

As noted in the comments, I cannot set the default value of the parameter in code - this will crash ArcCatalog when publishing the service. If I do not set the value of that parameter in the code, then run the tool in ArcGIS Desktop and point that paramter to the appropriate *.sde file, the geoprocessing result will publish fine.

The problem I get now is that the path to the test.sde connection is being rewritten during the publishing process. If I set the geoprocessing tool's messaging level to 'Info', I will see the following error when I attempt to submit a job to the GPServer:

esriJobMessageTypeError: ERROR 000732: TestWorkspace: Dataset C:\Users\Administrator\AppData\Local\ESRI\Desktop10.1\Staging\arcgis on localhost_6080 (admin)\63\v101\test.sde does not exist or is not supported

#
 

The folder that contains the original sde file (C:/Data) and the geodatabase that the connection file points to are both registered as data stores on the server, so I don't know why the path to the sde connection file is being rewritten to point to a staging folder. Does anyone know what it is that I'm missing?

[KevinHibma]

Thanks for reporting.
I can reproduce this and will get a bug logged for it.

Just on the quick testing I've done: it seems specific to executing this workflow in ArcCatalog.
If I do the same inside ArcMap it works fine - can you try your workflow there?

thanks again.

[Anonymous User]

Hi Kevin,

Thanks for the reply.  You are correct, the first problem I note (ArcCatalog crashing) is not an issue if I publish using ArcMap instead.

However...I still get the problem I note in my second post - the paths to SDE connection are being rewritten to point to a staging folder, which will no longer exist after the service is published.

[KevinHibma]

When I publish from ArcMap, and go find the PTY in the Input directory, I have the following inside:

# Esri start of added variables
import os, arcpy
g_ESRI_variable_1 = os.path.join(arcpy.env.packageWorkspace,u'ne2k864_DB_auth.sde')
# Esri end of added variables

#........
    def execute(self, parameters, messages):
        """The source code of the tool."""
        
        wrkspc = g_ESRI_variable_1
        return

This is correct per the publishing process. My SDE connection file lives at:
c:\arcgisserver\directories\arcgissystem\arcgisinput\Tool.GPServer\extracted\v101
The v101 folder is the packageworkspace.

Are you using 10.1 final or 10.1 sp1?
I'm not sure how you're getting the staging directory pushed into the PYT. Thats definitely wrong, but I can't reproduce that.

Side note to your first point : a workspace isn't a supported input type with GP Services. We re-map that to constant (basically hardcode it). We do support "string", you could manually write that up. Without knowing your requirements, the only general statement is most of the time, workspaces arent necessary as a parameter visible to users. If any changes to the workspace need to be done, its best done behind the scenes.
Do you have a full PYT you could share (just so I can get a better idea what you're doing)? If you dont want to post here, you can email it to me: khibma@esri.com

thanks

[Anonymous User]

Hi Kevin,

It looks like the root of the problem is the presence/absence of the unicode 'u' modifier on the strings.  If I include that, I can set a variable to u"C:/Data/test.sde" or os.path.join(u"C:/Data",u"test.sde"), and publish fine with ArcMap (ArcCatalog still crashes).  If I leave out the 'u' character, ArcMap will crash just as badly as ArcCatalog when I attempt to publish the result.  After publishing successfully, I also find that path to the sde connection file gets translated to a location on the server, such as "C:\arcgisserver\directories\arcgissystem\arcgisinput\VGO\Test.GPServer\extracted\v101\test.sde"

I have been doing this testing in ArcGIS 10.1 SP1, and I hadn't posted a complete toolbox previously, since all that is needed to demonstrate the problem in my environment is a couple lines of code in the execute function .  However, an example of what I'm currently testing is attached.

So, aside from the bug that affects ArcCatalog, should I have known that I need to use unicode strings for this?

I still find that other strings get caught unexpectedly.  For example, I might want to save a prefix for feature class names that will be used when accessing them from the geodatabase.  For example:

myGeodatabase = os.path.join(u"C:/Data","test.sde")

featureClassPrefix = u"/test.sde."  ### Sometimes this will get replaced with a path that points to the staging folder...sometimes ArcMap crashes entirely...sometimes it's left untouched.

pointFeatureClass = myGeodatabse+featureClassPrefix+u"TestPoints"

I'm not able to consistently produce the same problem, but basically, I have to be very careful that I don't specify any string values that may be interpreted as a connection file.  If I substitute u'/test.sde' with u'/'+u'test.sde', the problem goes away.  Or if I rename the connection file so that it is not the same name as the underlying database (e.g., name the file test_default.sde), then the problem also will go away in that scenario.

It might be worth noting also that the connection file and the pyt file are in the same folder on the disk.

[KevinHibma]

I dont think I have all the answers to your questions, but I can say this:

When writing paths, you don't need the 'u'. (as long as they're written correctly - correctly being forward slash or double backslash or done with 'u' or 'r')
If you're building up paths I strongly encourage you to use the os.path.join function and never the "+" to add up paths. I cannot guarantee that when scanning your script for required data for publishing/packaging that strings (paths) built up with + will be caught. We can and do work quite well with os.path.join

I'm not sure if you've seen this help topic, if not I'd give a quick read:
http://resources.arcgis.com/en/help/main/10.1/#/Authoring_geoprocessing_tasks_with_Python_scripts/00...

The short of it: if we encounter project data in your script, we evaluate that against the datastore to either copy or not.
The following code works fine.

>>> import os
>>> start = "sde.sde"
>>> end = ".fc1"
>>> sdeServer = "c:/mydata/somesde.sde"
>>> print os.path.join(sdeServer, start + end)
c:/mydata/somesde.sde\sde.sde.fc1

However, if you start mixing in slashes with your strings, you'll see what happens when they get put together with os.path.join

>>> start2 = "/test.sde"
>>> print os.path.join(sdeServer, start2 + end)
/test.sde.fc1

This stackexchange answer describes what slashes do inside an os.path.join statement : http://stackoverflow.com/questions/1945920/os-path-join-python
So anytime that slash is encountered in os.path.join, its like resetting the path. Does this clear it up at all?


Regardless of all this, it shouldn't crash : I assigned the bug to a developer and we'll evaluate for 10.1.2
(NIM087309 - Publishing or sharing a PYT from ArcCatalog causes a crash when referencing a .SDE workspace)

Thanks for including your workflow. I'll take a look to make sure we're catching everything.

[KevinHibma · Accepted solution]

Mike,

We've debugged this down further and think we have an understanding of the problem.

If you still have a version of your toolbox before hacking it up to get around this problem, you can try this which should probably work:

In ArcCatalog, goto Geoprocessing > Environments.
Set a value for the Current Workspace (under the Workspace heading)
You can set this to a folder or a fGDB.

Now when publishing or packaging it should work.

Again, my apologies for the inconvenience. We're working on a fix that you'll hopefully see in service pack 2.

[Anonymous User]

I've got another dimension to add to this.  If I have just a plain string that I'm using to split a string with slashes, it gets interpreted as a reference to the entire hard drive:

import mimetypes
mtype = mimetypes.guess_type(filename)[0].split("/")

If I comment out the code with with the slash, the service will publish fine.  This is in ArcMap, with a default gdb set in the environment settings.

[Anonymous User]

For anyone else that comes across this thread with a similar problem, the following solution was suggested to me for splitting a string with slash characters:

import mimetypes
mtype = os.path.split(mimetypes.guess_type(filename)[0])

Thanks for all the help.