source ArcGIS Server cannot be found

[imbGISTeam]

In an ArcGIS for Server (10.3.1) cluster running on three VMs (Windows Server 2008R2 standard), during the replication process we are frequently encountering an error that causes the server to become unresponsive. The error message we see on the server is "source ArcGIS Server cannot be found". Has anyone encountered this issue and found a solution?

[DanielUrbach]

I'm assuming the error message you mentioned is from the Windows Event Viewer logs.

Are there any log entries in the ArcGIS Server site that indicate with machine is having problems?

I believe this issue may be related to the Windows user that is running the ArcGIS Server service, specifically its ability to 'log on as a service'.  A change in group/local policy can cause this right to be revoked from that user.

You may want to confirm that the account running the service has the 'log on as a service' right on that server, then attempt to restart the service. 

[imbGISTeam]

Daniel, you are correct. The error message mentioned is from the Windows Event Viewer logs. I failed to include that this is an intermittent problem. We run ETLs every six hours that update the file geodatabases in our data-stores. Typically these updates replicate throughout the cluster without issue, but once or twice a week we'll experience a failure.

A review of the Server Event log includes the following:

<Msg time="2015-06-12T14:03:21,597" type="SEVERE" code="9016" source="Rest" process="2952" thread="20" methodName="" machine="RVAVPAPP26" user="" elapsed="">Error getting service.</Msg>

<Msg time="2015-06-12T14:03:21,601" type="SEVERE" code="9003" source="Rest" process="2952" thread="20" methodName="" machine="RVAVPAPP26" user="" elapsed="">Unable to process request. Service Dev/PeopleGroupsOrg_Dev/MapServer not found </Msg>

<Msg time="2015-06-12T14:03:29,568" type="SEVERE" code="9016" source="Rest" process="2952" thread="20" methodName="" machine="RVAVPAPP26" user="" elapsed="">Error getting service.</Msg>

<Msg time="2015-06-12T14:03:29,568" type="SEVERE" code="9003" source="Rest" process="2952" thread="20" methodName="" machine="RVAVPAPP26" user="" elapsed="">Unable to process request. Service Dev/PeopleGroupsOrg_Dev/MapServer not found </Msg>

Once the server is restarted the processes tend to run smoothly for several days, then we'll experience another failure.

[DanielUrbach]

Is this issue always with the RVAVAPP26 machine?  It may be that something is preventing this machine from retaining contact with the config-store of the site, or causing that connection to periodically break.

I would recommend opening a ticket with Esri Support Services if that is available to you, as there are many reasons why this could happen and it may take a bit more intensive troubleshooting than we can do over a forum like this!

[MarcusAndersson]

Hi!
Old thread, but did this ever get solved? 
I'm experiencing the same issues right now (Enterprise 10.8.1, Windows Server 2019)..

Thankful for any hints about this! 

BR,
Marcus

[GISCloud]

We encountered the same problem and the same software version as you, but we are using Red Hat Enterprise 9.2 system.
After we released the scene service (slpk), it could not be loaded in the web browser, but it could be loaded normally on the desktop. Two-dimensional map slicing did not have this problem.
The error log is as follows:

<Msg time="2024-04-13T01:54:47,591" type="WARNING" code="9016" source="Rest" process="4223" thread="1" methodName="" machine="server.xxxx.com" user="" elapsed="" requestID="9fef31f7-8d07-42d8-92aa-f97935cc716f">Error getting service.</Msg>
<Msg time="2024-04-13T01:54:47,591" type="WARNING" code="9003" source="Rest" process="4223" thread="1" methodName="" machine="server.xxxx.com" user="" elapsed="" requestID="9fef31f7-8d07-42d8-92aa-f97935cc716f">Unable to process request. Service Hosted/baoli_v1_9/SceneServer not found </Msg>