nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Published GP Service keeps copying data to server

ZacharyHart

I almost put this in the AGS 10.1 General section, but landed here.

So just like the title says, not matter how I try, when I publish my GP service (results from a model) the relevant SDE data gets copied to the server down in \\servername\arcgisserver\directories\arcgissystem\arcgisinput.

The SDE DB is registered with the GIS server and no warning/messages appear about copying data at the time of publishing. I've been through tech support and the incident was escalated. The model is very simple....essentially moving selected features from one FC to another via append. The GP service runs successfully on the server but appends to that fGDB copy that resides on the server. Is this expected behavior?

Find more posts tagged with

Accepted answers

All comments

MetralAlexis

Ok, I've found a solution for the next ones facing that issue : you need to publish your service from ArcMap (don't ask me why)

If I publish my service from ArcCatalog, it doesn't work, upload the data with no warning in the v101 folder and is useless
If I publish my service from ArcMap, it works, upload the sde connection only and properly point to my data

Alex.

MetralAlexis

Hi Zachary,

I've got exactly the same issue on AGS 10.3.1, exactly the same and I'm knocking my head against the wall.

No warning with data store properly registered and ArcGIS Server that keeps uploading the gdb in the v101 folder with no warning about data that will be uploaded.

Were you able to fix that issue ?

Thanks

MIchaelHubbard

I think I have found the solution to this, and it is a stupid one. I was having the same problem. When I created the map document that I am publishing, I told an image file in the map document to "Save Image as part of document". This image file was located on a different server. I copied the image to a folder that I had registered as a data source and published again through ArcMap and it worked!

Perhaps I am making this too simple and it will not work for you but I hope it does. This was a ridiculous pain!

Best of luck!

Mike
GIS Analyst
Arapahoe County Government

ZacharyHart

I can't reproduce this (I'm responsible for testing NIM087822 and assigning to a developer).

If you have a minute can you look at the attached word doc. Is there anything you've done differently compared to my workflow?
If there is indeed a bug here (as I've done something differently than you), I'd like to get it addressed for the next service pack. However that window is quickly closing so any information you can provide would be great.

The analyst was able to reproduce this. The there is no warning that the data will be copied to the server; the sde connection file is not copied to the server, an extract of the SDE data is copied to the server arcgisinput directory as a fGDB.

Permissions are set for the local arcgis account (i even reset the password to all arcgis accounts to be sure this wasn't an issue); and i tried both registering the db server directly, the SDE connection file as you show, moving the SDE connection file to several different physical locations to see if registering it then would return different results etc etc. I looked from the SQL mgt studio side of things with the analyst too to ensure the local account was being used correctly by SQL Express and still no luck. Perhaps you can contact the analyst who worked on this, we spent several weeks trying to diagnose this so he probably has extensive notes.

My suspicion had been that there was an environment setting error, but the analyst did not believe that was the case. The tool worked fine locally, and if we modified the published tool on the server to use the correct data sources it would work then, but as i recall, only work when consumed via a web application, not desktop.

KevinHibma

I can't reproduce this (I'm responsible for testing NIM087822 and assigning to a developer).

If you have a minute can you look at the attached word doc. Is there anything you've done differently compared to my workflow?
If there is indeed a bug here (as I've done something differently than you), I'd like to get it addressed for the next service pack. However that window is quickly closing so any information you can provide would be great.

sqlexpress_gp_service.docx

ZacharyHart

[#NIM087822 GP service using feature class from sql express workgroup sde will have data copied to the server even the gdb was registered with server data store. ]
[#NIM087819 Feature service shows no editable layer error when the feature class was from workgroup sde (sqlexpress) with OS authentication ]

That is right, it basically makes feature service not editable when using sql express in server 10.1.

This is rather profound IMO.

achapapkowski

If you install 10.1 SP1, you can set the server to block all data copying. Check out this: http://resources.arcgis.com/en/help/main/10.1/#/Example_Prevent_data_copying_at_publish_time/01540000061p000000/

This might help your issue.

To do this without python, you can follow these instructions: http://resources.arcgis.com/en/help/main/10.1/#/Disabling_automatic_data_copying_when_publishing_to_the_server/01540000060n000000/

ZacharyHart

Any other ESRI staff want to take a shot at this? its been weeks now and I still dont have an answer from the escalated incident.

ZacharyHart

UPDATE:

soo another phone session this afternoon, and tried many things before that. Here's what i've got so far:

No matter how hard I try, when publishing a GP service the data copies to the server
When you review the published copy of the model on the server it refrences the server's copy of the data
You can modify the model to reference the correct SDE connection to your feature classes and restart the service and it will work correctly
The only way the modified GP service will work is against a map service referencing the same SDE features; if you try the published GP service against the local connection to the SDE data, it completes without warning and no results are depicted/appended. This was replicated via executing the GP task at REST; if the feature being accessed the task ran but produced no results; if no connections were active to the feature the rest service was able to write/append to the SDE FC just fine.
I can publish a map service with feature access capabilities against the same data set and it publishes without copying any data just fine
i can access the map service and run the gp service against it and it works just fine (see above)
If I attempt to add the feature service from the TOC to an arcmap session it instantly crashes with no error
If I expose the feature service via a web application it works fine (last I checked anyway)

So this may need to be moved over to the arcgis server 10.1 forum. I dont know but i'm ready to just give up...it will be nearly a week since it was escalated and nearly 2 weeks since the initial problem was discovered and the incident was initiated. I'm just so completely at a loss.

ZacharyHart

No... it will only copy the data if it doesnt find a match in the datastore.

If you can publish a map service and that data doesnt get copied, then the datastore is setup correctly.
I've never heard anyone report this particular issue you're running into: not warned about data being copied, but then the data is copied.

You said you logged a support incident? Whats the incident #? I want to look up the notes and see if anything stands out.

sent you a PM

KevinHibma

I can read till 'regardless'.....what a drag.

ok, now with a 'real' browser i can read the rest of what Kevin posted. It seems to suggest that at 10.1 GP services copy data to the server no matter what?
Even then, my problem is still with the published model. The output references that file gdb published to the server, not my SDE connection.

No... it will only copy the data if it doesnt find a match in the datastore.

If you can publish a map service and that data doesnt get copied, then the datastore is setup correctly.
I've never heard anyone report this particular issue you're running into: not warned about data being copied, but then the data is copied.

You said you logged a support incident? Whats the incident #? I want to look up the notes and see if anything stands out.

ZacharyHart

I can read till 'regardless'.....what a drag.

ok, now with a 'real' browser i can read the rest of what Kevin posted. It seems to suggest that at 10.1 GP services copy data to the server no matter what?
Even then, my problem is still with the published model. The output references that file gdb published to the server, not my SDE connection.

ZacharyHart

Kevin, trying to read more on the article you provided....can only see the top of the section you refrence. Sadly, my Droids browser never let's me scroll down on ESRI help entries....and only ESRI help entries....no other site....oh well.

ZacharyHart

Yes, same service and same location, same sde db all publish as map and feature services.....the exact same data! The gp publishes, no warning of copying data, but it does! copy the data to the server and modifies the output of the model to that input directory. 2 weeks with tech support, escalated and still not working.

Simple data set running against sde behaves the same way.

DaleHoneycutt

Just a shot in the dark here: take a look at this topic A quick tour of publishing a geoprocessing service and scroll down until you find the section titled "Small output datasets". It explains how output data is copied to the server in order to support map services. This may or may not by your situation.

KevinHibma

Can you make the simplest of map services with 1 or 2 of the same layers your GP tool uses and try to publish that?

Map and GP services follow the same rules, there could be a different code path they go down.

If map service copies the data then it at least shows its a wider issue or if not, then specific to GP.

ZacharyHart

interesting. So for the heck of it, i changed my registered SQL Express DB connection information to just the name of the instance... without \sqlexpress

I can see the correct gdb and it validates fine. Now i re-run my model and when i try to publish the results i'm warned about data getting copied to the server.

still no joy. For the life of me, i cannot figure out why its copying the data...especially with no warning!

ZacharyHart

interesting. So for the heck of it, i changed my registered SQL Express DB connection information to just the name of the instance... without \sqlexpress

I can see the correct gdb and it validates fine. Now i re-run my model and when i try to publish the results i'm warned about data getting copied to the server.

ZacharyHart

registered via the gis server>server properties>data store. only talking about 2 FC here, both from the same GDB; i dont see how folders come into play.

I've done a bit of arcpy scripting successfully but dont have the time to get the parameters lined up when i convert to python (this model also uses in memory which i dont think is supported for python either).

when publishing there is no warning about data getting copied over, but its down there in the directories!.

achapapkowski

How did you register the sde connection (There are 2 ways)? Do you have the associated folders registered as well?
If you do the same function via a python tool, not a model, do you see the same results?
Thanks

ZacharyHart

Already done...no dice.

achapapkowski

Did you register your sde connection with server?

That can cause data to copy.

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.