topic Re: Published GP Service keeps copying data to server in Geoprocessing Questions

Published GP Service keeps copying data to server

ZacharyHart — Tue, 18 Dec 2012 14:25:08 GMT

I almost put this in the AGS 10.1 General section, but landed here.

So just like the title says, not matter how I try, when I publish my GP service (results from a model) the relevant SDE data gets copied to the server down in \\servername\arcgisserver\directories\arcgissystem\arcgisinput.

The SDE DB is registered with the GIS server and no warning/messages appear about copying data at the time of publishing. I've been through tech support and the incident was escalated. The model is very simple....essentially moving selected features from one FC to another via append. The GP service runs successfully on the server but appends to that fGDB copy that resides on the server. Is this expected behavior?

Re: Published GP Service keeps copying data to server

AndrewChapkowski — Wed, 19 Dec 2012 15:22:37 GMT

Did you register your sde connection with server?

That can cause data to copy.

Re: Published GP Service keeps copying data to server

ZacharyHart — Wed, 19 Dec 2012 16:23:42 GMT

Already done...no dice.

Re: Published GP Service keeps copying data to server

AndrewChapkowski — Wed, 19 Dec 2012 16:29:18 GMT

How did you register the sde connection (There are 2 ways)? Do you have the associated folders registered as well?
If you do the same function via a python tool, not a model, do you see the same results?
Thanks

Re: Published GP Service keeps copying data to server

ZacharyHart — Wed, 19 Dec 2012 16:51:22 GMT

registered via the gis server>server properties>data store. only talking about 2 FC here, both from the same GDB; i dont see how folders come into play.

I've done a bit of arcpy scripting successfully but dont have the time to get the parameters lined up when i convert to python (this model also uses in memory which i dont think is supported for python either).

when publishing there is no warning about data getting copied over, but its down there in the directories!.

Re: Published GP Service keeps copying data to server

ZacharyHart — Wed, 19 Dec 2012 17:49:11 GMT

interesting. So for the heck of it, i changed my registered SQL Express DB connection information to just the name of the instance... without \sqlexpress

I can see the correct gdb and it validates fine. Now i re-run my model and when i try to publish the results i'm warned about data getting copied to the server.

Re: Published GP Service keeps copying data to server

ZacharyHart — Wed, 19 Dec 2012 18:51:34 GMT

interesting. So for the heck of it, i changed my registered SQL Express DB connection information to just the name of the instance... without \sqlexpress

I can see the correct gdb and it validates fine. Now i re-run my model and when i try to publish the results i'm warned about data getting copied to the server.

still no joy. For the life of me, i cannot figure out why its copying the data...especially with no warning!

Re: Published GP Service keeps copying data to server

KevinHibma — Wed, 19 Dec 2012 20:48:01 GMT

Can you make the simplest of map services with 1 or 2 of the same layers your GP tool uses and try to publish that?

Map and GP services follow the same rules, there could be a different code path they go down.

If map service copies the data then it at least shows its a wider issue or if not, then specific to GP.

Re: Published GP Service keeps copying data to server

DaleHoneycutt — Wed, 19 Dec 2012 20:56:40 GMT

Just a shot in the dark here: take a look at this topic A quick tour of publishing a geoprocessing service and scroll down until you find the section titled "Small output datasets". It explains how output data is copied to the server in order to support map services. This may or may not by your situation.

Re: Published GP Service keeps copying data to server

ZacharyHart — Wed, 19 Dec 2012 23:10:18 GMT

Yes, same service and same location, same sde db all publish as map and feature services.....the exact same data! The gp publishes, no warning of copying data, but it does! copy the data to the server and modifies the output of the model to that input directory. 2 weeks with tech support, escalated and still not working.

Simple data set running against sde behaves the same way.

Re: , Published GP Service keeps copying data to server

ZacharyHart — Thu, 20 Dec 2012 00:29:54 GMT

Kevin, trying to read more on the article you provided....can only see the top of the section you refrence. Sadly, my Droids browser never let's me scroll down on ESRI help entries....and only ESRI help entries....no other site....oh well.

Re: Published GP Service keeps copying data to server

ZacharyHart — Thu, 20 Dec 2012 00:35:32 GMT

I can read till 'regardless'.....what a drag.

ok, now with a 'real' browser i can read the rest of what Kevin posted. It seems to suggest that at 10.1 GP services copy data to the server no matter what?
Even then, my problem is still with the published model. The output references that file gdb published to the server, not my SDE connection.

Re: Published GP Service keeps copying data to server

KevinHibma — Thu, 20 Dec 2012 15:10:30 GMT

I can read till 'regardless'.....what a drag.

ok, now with a 'real' browser i can read the rest of what Kevin posted. It seems to suggest that at 10.1 GP services copy data to the server no matter what?
Even then, my problem is still with the published model. The output references that file gdb published to the server, not my SDE connection.

No... it will only copy the data if it doesnt find a match in the datastore.

If you can publish a map service and that data doesnt get copied, then the datastore is setup correctly.
I've never heard anyone report this particular issue you're running into: not warned about data being copied, but then the data is copied.

You said you logged a support incident? Whats the incident #? I want to look up the notes and see if anything stands out.

Re: Published GP Service keeps copying data to server

ZacharyHart — Thu, 20 Dec 2012 15:55:38 GMT

No... it will only copy the data if it doesnt find a match in the datastore.

If you can publish a map service and that data doesnt get copied, then the datastore is setup correctly.
I've never heard anyone report this particular issue you're running into: not warned about data being copied, but then the data is copied.

You said you logged a support incident? Whats the incident #? I want to look up the notes and see if anything stands out.

sent you a PM

Re: Published GP Service keeps copying data to server

ZacharyHart — Thu, 20 Dec 2012 20:07:27 GMT

UPDATE:

soo another phone session this afternoon, and tried many things before that. Here's what i've got so far:

No matter how hard I try, when publishing a GP service the data copies to the server

When you review the published copy of the model on the server it refrences the server's copy of the data

You can modify the model to reference the correct SDE connection to your feature classes and restart the service and it will work correctly

The only way the modified GP service will work is against a map service referencing the same SDE features; if you try the published GP service against the local connection to the SDE data, it completes without warning and no results are depicted/appended. This was replicated via executing the GP task at REST; if the feature being accessed the task ran but produced no results; if no connections were active to the feature the rest service was able to write/append to the SDE FC just fine.

I can publish a map service with feature access capabilities against the same data set and it publishes without copying any data just fine

i can access the map service and run the gp service against it and it works just fine (see above)

If I attempt to add the feature service from the TOC to an arcmap session it instantly crashes with no error

If I expose the feature service via a web application it works fine (last I checked anyway)

So this may need to be moved over to the arcgis server 10.1 forum. I dont know but i'm ready to just give up...it will be nearly a week since it was escalated and nearly 2 weeks since the initial problem was discovered and the incident was initiated. I'm just so completely at a loss.

Re: Published GP Service keeps copying data to server

ZacharyHart — Thu, 03 Jan 2013 13:00:35 GMT

Any other ESRI staff want to take a shot at this? its been weeks now and I still dont have an answer from the escalated incident.

Re: Published GP Service keeps copying data to server

AndrewChapkowski — Fri, 04 Jan 2013 12:20:43 GMT

If you install 10.1 SP1, you can set the server to block all data copying. Check out this: http://resources.arcgis.com/en/help/main/10.1/#/Example_Prevent_data_copying_at_publish_time/01540000061p000000/

This might help your issue.

To do this without python, you can follow these instructions: http://resources.arcgis.com/en/help/main/10.1/#/Disabling_automatic_data_copying_when_publishing_to_the_server/01540000060n000000/

Re: Published GP Service keeps copying data to server

ZacharyHart — Mon, 07 Jan 2013 11:02:27 GMT

[#NIM087822 GP service using feature class from sql express workgroup sde will have data copied to the server even the gdb was registered with server data store. ]
[#NIM087819 Feature service shows no editable layer error when the feature class was from workgroup sde (sqlexpress) with OS authentication ]

That is right, it basically makes feature service not editable when using sql express in server 10.1.

This is rather profound IMO.

Re: Published GP Service keeps copying data to server

KevinHibma — Mon, 07 Jan 2013 16:20:03 GMT

I can't reproduce this (I'm responsible for testing NIM087822 and assigning to a developer).

If you have a minute can you look at the attached word doc. Is there anything you've done differently compared to my workflow?
If there is indeed a bug here (as I've done something differently than you), I'd like to get it addressed for the next service pack. However that window is quickly closing so any information you can provide would be great.

Re: Published GP Service keeps copying data to server

ZacharyHart — Mon, 07 Jan 2013 17:44:01 GMT

I can't reproduce this (I'm responsible for testing NIM087822 and assigning to a developer).

If you have a minute can you look at the attached word doc. Is there anything you've done differently compared to my workflow?
If there is indeed a bug here (as I've done something differently than you), I'd like to get it addressed for the next service pack. However that window is quickly closing so any information you can provide would be great.

The analyst was able to reproduce this. The there is no warning that the data will be copied to the server; the sde connection file is not copied to the server, an extract of the SDE data is copied to the server arcgisinput directory as a fGDB.

Permissions are set for the local arcgis account (i even reset the password to all arcgis accounts to be sure this wasn't an issue); and i tried both registering the db server directly, the SDE connection file as you show, moving the SDE connection file to several different physical locations to see if registering it then would return different results etc etc. I looked from the SQL mgt studio side of things with the analyst too to ensure the local account was being used correctly by SQL Express and still no luck. Perhaps you can contact the analyst who worked on this, we spent several weeks trying to diagnose this so he probably has extensive notes.

My suspicion had been that there was an environment setting error, but the analyst did not believe that was the case. The tool worked fine locally, and if we modified the published tool on the server to use the correct data sources it would work then, but as i recall, only work when consumed via a web application, not desktop.