Orphaned ArcSOC.exe

JustinHunter · ‎09-14-2013

Just a few moments ago I came across a curious issue - the over-writing of one of my services continued to fail. This isn't the first time this has happened to me - and last time I fixed it when I re-ran the ArcGIS Server configuration - the one that you specify which user ArcGIS Server is running under. I kept getting the error...

SEVERE

Sep 14, 2013 2:23:26 PM

Failed to create the service.: Unable to delete service input folder.

System/PublishingTools.GPServer

Let me continue with a back story. Last night I was fiddling around doing queries - for some reason through the REST interface my service will -never- return if I do an objectid lookup. I have to put OBJECTID=n into the WHERE instead. Me doing this last night caused one of my services to be become unstable - showing one constantly being in use. I shrugged - I know ArcGIS Server will kill off a service that's in use for too long.

However this morning that same one was still in use. I stopped my service and looked at task manager - the ArcSOC.exe for that service did not stop.

Then I decided to try to overwrite the service. Failure - however the service was now no longer part of my list of services on the manager portal. I was confused - I checked folder permissions and everything was in check. It turns out that orphaned ArcSOC.exe was holding onto that folder. I understand that.

What I do NOT understand is how I even GOT an orphaned ArcSOC. What about the automatic clean-up? Why did it miss this one? Perhaps I am misunderstanding "The maximum time a client can use a service:" - which I assumed was ticking down on the "Instances in Use: 1" - with the recycling options, which are set to 24 hours.

However, even STOPPING the service didn't kill off the ArcSOC, so I assume the recycling wouldn't have done it either.

Any advice would be fantastic. Sorry for the length. Try to fit as much as I can.

RichardWatson · ‎09-15-2013

I have seen this pattern before and believe it is an ArcGIS Server bug. Essentially, you have a boundary condition where ArcGIS Server has removed an instance of ArcSOC from management but yet the process did not cleanly exit. You can attach a debugger, such as WinDbg, to the hung process and attempt to figure out why it is hung. This starts to get deep fast.

Do you have a custom Server Object Extension? If so then post back and I'll share a good trick with you with regards to how I addressed this issue.

If there is no custom SOE then the only way to resolve this is that you have to file a reproducible incident with ESRI support.

JustinHunter · ‎09-15-2013

My service does have an SOE that I built attached to it, however running it was not the cause of the original hanging - if that makes a difference.

RichardWatson · ‎09-15-2013

The way that I solved this problem was when the SOE is Shutdown I start a background thread. What the thread does is to wait some amount of time (say 5 minutes) and then it kills the process. What this does is to give the process time to cleanly exit. If it hangs then death is forced. Doing this stops the zombie ArcSOC processes.

JustinHunter · ‎09-16-2013

This sounds like a custom SOE I could build that has no purpose other than to ensure the process does eventually end. What I'm worried about is that these processes are kept track of somewhere and my SOE that kills it may cause issues - such as ArcGIS Server thinking there are 5 processes already running when in fact only 4 are.

Thanks for your suggestion. I'm going to duplicate this and bring it to ESRI's attention.