ERROR: “ArcGIS Server Site is currently being configured by another administrative operation. Please try again later.”
I got the error shown in the screenshot below
What might be the issue here?
Thank you
Best
Jamal
I've seen this before when I've tried to change the service directories for a site and something and it stuck for some reason and I couldn't do anything. Did this just randomly start to occur? or did something happen to cause this error coming up.
I think restarting the ArcGIS Server service resolves the issue.
Thanks Shaun.
That has occurred after upgrading the ArcGIS Server from 10.2 to 10.2.2.
I think my approach in the upgrade is wrong. I have uninstalled the 10.2 and then installed the 10.2.2. This is why I got numerous issues
Jamal,
I have encountered almost the exact same issue you describe. I upgraded our server from 10.2 to 10.2.2. I installed 10.2.2 in-place though (did not uninstall 10.2 first) so I don't think that your approach is the issue. Post install, many of my services are stuck in a "Stopping..." state. I eventually got them to stop, but when I try to start the whole machine (or a single service) I get the "ArcGIS Server is currently being configured by another administrative operation..." message. I am doing this upgrade on our development server to iron out any kinks before I upgrade our production ArcGIS Server site. I am determined to get to the bottom of this.
Scott
Scott,
The exact cause for "Stopping..." services and the "ArcGIS Server is currently being configured by another administrative operation..." message are hard to attribute to any one thing, but you may be able to un-deadlock your ArcGIS Server and services by removing any .glock files in the "locks" directory in the config-store.
In my experience, I have never had an ArcGIS Server upgrade go smoothly, even for seemingly simple service packs. There was always something with Stopping services or the like. Things always ironed themselves out after enough reboots and a bit of attention to that "locks" directory between restarts of ArcGIS Server.
Many thanks Erin and Scott for the input,
The issue here is not why this error comes up but it is the way it can be resolved.
Is there a robust method to resolve it in case it is produced?
Hello Erin,
Can you tell what is the influence of deleting the .glock files? I have a military client who wants to delete these files but is afraid that it will ruin something in his operational system.
Alternatively, is creating of these files reproducible?
Thanks a lot,
Miri Eshel
Hi Miri,
In my experience, deleting .glock files did not have unintended consequences on the system/ArcGIS Server, though this was always done as a last resort and in a surgical fashion (i.e. only the .glock file for a single locked-up service was deleted). Once these files are removed, it is possible to administer the related services and return them to a usable state.
You may run into a situation where the ArcGIS Server Java process (javaw.exe) has hooks into the .glock file that will prevent you from deleting it. In that case, you will either have to restart ArcGIS Server (to release its hooks in the file), or use "kludgy" tools to force that same release. Of course, if you are running in a production environment then restarting ArcGIS Server is less of an option and you may have to turn to tools like this one.
On a related note, there is a bug number for this problem (NIM100965: "Starting a service with 0 minimum instances causes the service locks not to release if service is consumed while it is starting"), and the Issues Addressed List reports it as being fixed in 10.3.
I was able to reproduce this issue on a pristine server and install of ArcGIS Server (10.2.1) by creating a high-isolation Map Service with 0 minimum instances and then rapid-fire reloading the service's WSDL URL while the service started.
Hope this info is helpful to you.
Hi Jamal!
We had the same problem under ArcGIS Server 10.2. We adjusted the folder priviliges to full access for the local user "arcgis" for the following folders: arcgisserver\config-store\; arcgisserver\directories\; arcgisserver\ServicesData\.
After restarting ArcGIS Server service all services had been started.
Regards,
Peter
So didn't notice any comments regarding if anyone was using Clustered Servers or if you are getting this in a single server environment.
I can say that we have several environments with as few as 2 and up to 6 clustered servers and have seen this many times. We do keep a number of our 300+ services running with an initial instance count of zero to conserve on full time resource usage. In general when we have seen this it's been a result of issues propagating changes across the multiple clustered servers. Something as simple as starting or stopping services is a command that has to occur across all the services in the cluster that host the service. It seems pretty common for 1 or more of the servers to hang up or not properly execute the commands. When we see this our easy fix has been to go into Server Manager and simply stop machines one at a time and re-verify the status then repeat until it is resolved. Sometimes stopping a single machine is all it takes and others we have had to go down to a single server.
Once the server that was hanging or not providing a valid response back has been stopped and this error is no longer appearing you can simply restart the server in the cluster and it will join back in with the appropriate change made (in this example the services is correctly stopped or started but this issue has been seen during configuration changes being made, etc. with the worst issues of this occurring when updating certificates).
Hope that helps some who are clustering at least.
Very Respectfully,
Scott Fierro
I would actually recommend stopping the ArcGIS Server service and then deleting all of the .rlock files found in the various sub-directories of config-store as well as the .glock file. Sometimes those .rlock files do not get removed properly and lock a file which will cause that error.
i've found that when clustered, it is the appliance (disk arrays) where the shared config store is that causes these issues (e.g. EMC Isilon) and not ESRI Software.
Hello,
I am facing same issue after upgrade 10.3.1 to 10.4.
After this issue i have need to restart ArcGIS server services and then start the stopped/stopping services.
Please suggest why this type of issue happen.
Hi,
We had problems with publishing map services after upgrading to 10.4. When a map service failed to publish it would leave lock files behind which could only be cleared after a restart of ArcGIS Server.We have resolved this issue by setting all our map services min instances to 0. I suspect the new publishing patch may also help, but I've yet to test it.
Cheers,
Sarah Noakes
Cornwall Council
Hello Sarah,
The new publishing patch does not resolve this issue.
When I have received or facing this error, then only AGS process restarts is correct and useful option at that time.
Thanks!
I had this issue on a production server so restarting the service or machine was not an option. I managed to resolve it by deleting the lock files for the specific service in:
The lock file itself was locked by another process (javaw.exe) so I had to use Process Expolorerto indentify the lock and force the delete. (This didn't cause me any problems, but things like this come with all the regular warnings of messing your system up, so beware. This was ArcGIS Server 10.4.1 on Windows Server 2008 R2 SP1)
To use process explorer:
Thank You - Worked like a charm! We have been struggling with this issue since 10.4.1
There is a publishing patch we are going to try next week to see if it fixes the actual issue. We just need to wait until we can do it off hours because it restarts all services.
For others: Process Explorer Can Be Download Here
Drew
That helped! The right-cilck->close step did not work for some reason, but instead you can find the Process ID in the Task Manager and just end the task before finding and deleting the little gremlin lock file.
Hi Jamal,
Were you ever able to find out what was causing the issue and how to resolve it?
Greetings all,
I have a test environment, with ArcGIS Enterprise 10.5 running on a VM. For some reason, I started getting the error which is the subject of this entry. So I followed the advice offered here and here. But to no avail. When I checked the Task Manager, I could see my three ArcGISServer.exe running, but no ArcSOC.exe s at all.
So I then tried what Erin Brimhall gave Scott on an entry above: "... but you may be able to un-deadlock your ArcGIS Server and services by removing any .glock files in the "locks" directory in the config-store." and sure enough, after deleting the lock files in the C:\arcgisserver\directories\arcgissystem\locks directory, and restarting the ArcGISServer process, all the ArcSOC.exe started coming back to life. After a little while, all services were running, and Server was once again usable.
Many thanks to Erin for the insight to the solution that worked for me. I much appreciate it.
Let us know if the same issue comes back.It's not a one time occurrence for me, its a life style since 10.4 I have found setting all services at a minimum instances of 0 helping a lot though.
It's crazy, more than 3-years now since this thread was first started and the .rlock/.glock issue continues to dog some ArcGIS Server admins. On the positive side, I'm glad the responses captured here by myself and others are still helping the community.
This method works.
Steps:
1)stop arcgis server services.
2)delete the lock files(*.glock and *.rlock ) in arcgisserver\config-store.
3) restart arcgis server service.
4)stop the pending stopping service and then start it.
This issue appeared in ArcGIS Server 10.2.2
Worked exactly as specified. Thanks!
I need to highlight that many errors in the ArcGIS Server requires stopping the services which really bother users for a while. Online users are shouting due to the number of times the ArcGIS Server is down. I would appreciate if this issue is fixed without the need to stop the ArcGIS Server. May be in the next releases!
Hello
I'm using an ArcGIS Server 10.5.1 on CentOS 7.2 and having the same issus.
In the arcgisserver\directories\arcgissystem\locks directory, I found file with lock.info extension.
This is the trace of the issue;
arcgisserver\directories\arcgissystem\locks directory
Here is an extract of the lock.info file
PID : 22070TRACE : 362 : JMX server connection timeout 362 java.lang.Object.wait(Native Method) com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168) java.lang.Thread.run(Thread.java:745)296 : RMI TCP Connection(51)-192.168.2.176 java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.FilterInputStream.read(FilterInputStream.java:83) sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$7/264670546.run(Unknown Source) java.security.AccessController.doPrivileged(Native Method) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745)273 : RMI TCP Connection(33)-192.168.2.176 java.lang.Object.wait(Native Method) com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:449) com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:227) com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:274) javax.management.remote.rmi.RMIConnectionImpl$4.run(RMIConnectionImpl.java:1270) javax.management.remote.rmi.RMIConnectionImpl$4.run(RMIConnectionImpl.java:1268) java.security.AccessController.doPrivileged(Native Method) javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1276) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346) sun.rmi.transport.Transport$1.run(Transport.java:200) sun.rmi.transport.Transport$1.run(Transport.java:197) java.security.AccessController.doPrivileged(Native Method) sun.rmi.transport.Transport.serviceCall(Transport.java:196) sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$7/264670546.run(Unknown Source) java.security.AccessController.doPrivileged(Native Method) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745)271 : RMI TCP Connection(32)-192.168.2.176 java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.FilterInputStream.read(FilterInputStream.java:83) sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$7/264670546.run(Unknown Source) java.security.AccessController.doPrivileged(Native Method) sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745)240 : Thread-115 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) com.esri.arcgis.discovery.logging.Logger$c.run(Logger$c.java:464)58 : Thread-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) com.esri.arcgis.discovery.logging.Logger$c.run(Logger$c.java:464)247 : Geronimo AppServer Starter java.io.FileInputStream.readBytes(Native Method) java.io.FileInputStream.read(FileInputStream.java:255) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.FilterInputStream.read(FilterInputStream.java:107) com.esri.arcgis.discovery.util.Pipe.run(Pipe.java:44)
There is a JMX connection timeout. but it seems to be cause by the logger
com.esri.arcgis.discovery.logging.Logger$c.run(Logger$c.java:464)
My log is on a NAS. Then I change the configuration of the log to put them locally
and restart my server.
My issue has been resolved !!!
I just need to check why logging on my NAS was an issue.
Note that there is ".lck" file on the logger directory.
Then I set the log on the previuos NAS directory
Stop arcgis server
Delete ".lck" file
Start arcgis server
And Bingo ! It works
Then deleting .lock, .rlock .gloc .wlock is not enougth : you should add .lck on log directory...
Technically speaking, using a network file location for your ArcGIS Server logs is not a supported configuration. See About specifying server log settings—ArcGIS Server Administration (Windows) | ArcGIS Enterprise
Specifically this line:
"The log path must be set to a local directory and exist on each GIS server participating in your ArcGIS Server site."
The .lck files are most likely generated by your NAS.
-Danny
I agree with you Daniel.
We'd experienced this issue before. We changed the Arcgis server directory to NAS & we kept facing down services until we changed the directory back to the local machine.
Best,
Alaa
More than likely, whomever administers or configured your NAS has OpsLocks enabled. Disable OpsLock and your issues will likely be gone.
I prefer using SAN for the superior performance of a NAS.
I had posed similar earlier in the thread. Isilon is a NAS appliance.
Thanks for your input Michael.
We'll try this.
Hi Way to disable opsTo Disable SMB2 and OPLOCK (Network Locking Errors) – Infusion Business Software
Users experiencing this issue should also consider logging cases with the NAS/SAN vendors. The essential issue here is that the device can't keep up with with the high IO between ArcGIS Server and the remote config-store.
Hi Randall,
In my case, I had, and it was brutal. Was multiple weeks of conference calls and deep diving and I was in a position with no luxury of time due to high business demands and limited resourcing. EMC (the vendor for Isilon), eventually figured out how to get the NAS running without AGS issues, however, I had created a clustered environment mirroring the setup but used EMCs vMAX disks (SAN) and the performance was 10 fold different. EMC failed to determine why the performance 'sucked' with the NAS in its configuration. I had to pull the plug when we had reached week four and several 'specialists' from the vendor and the possible weeks/months more internal process for Isilon changes (because it was used as a 'network' drive for many business groups) I went out of my way to resubmit an architectural change and introducing a single point of failure with the SAN setup. Far better than tinkering with that NAS let me tell you.
It is recommended to leave the configuration of a NAS appliance to a 'network' / IS / IT specialist. Not for a GIS 'anyone' to tinker with unless it is the GIS groups dedicated appliance and must be managed by them for whatever reason. NAS typically is shared with more than just the GIS group and is more of an organization IS backbone appliance.
Its great to 'know about it' but not recommended you act on it.
Scale-Out NAS for Unstructured & Big Data | Dell EMC Belgium
Přihlášení členové mohou přispívat, sledovat aktualizace a další. Jste tu noví? Zaregistrujte si bezplatný účet.
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.
