services missing

[WenliYang]

Somehow all services are missing, including the SampleWorldCities and the Utilities and System folders.

We use Windows 10.8.1 Enterprise in AWS.

The ArcGIS Server Software Authorization shows that the following are active: 

ArcGIS Server Advanced

ArcGIS Image Server

ArcGIS GeoEvent Server

Neither ArcGIS Server stop/start nor system reboot solved the problem.

Clearing the browser cache/history also didn't help.

The log shows that the problem occurred about 20 minutes after the "System/PublishingTools.GPServer" tool failed due not being able to copy data to server. Not sure if this is anything to do with the problem, but we set up the datastore and never copy data to server. 

Any help/tips will be greatly appreciated.

[jcarlson]

Can you still access your Portal and your Content, or are the content items missing as well?

[WenliYang]

I can still access portal and see the contents are displayed, but the sever is actually gone. If I try to show the content in the Online Viewer, I'll get an error message, for example:

The WMS service, https://arcgiscld.gesdisc.nasa.gov/server/services/SampleWorldCities/MapServer/WMSServer, cannot be added to the map. It is either not available or you have entered an invalid URL for the type of layer you want to reference.

[HenryLindemann · Accepted solution]

Hi @WenliYang,

First thing I would do is to check if the psychical services are still present 

C:\arcgis\arcgisserver\config-store\services

C:\arcgis\arcgisserver\directories\arcgissystem\arcgisinput

If these folders are still present then probably the folder permission was lost, 

Right click the arcgisserver folder go to security then advance and add your service account again also check if you service account is still working correctly.

You can also use the Configure ArcGIS Server Account to re-apply the folder permissions 

Hope it helps

Henry

[WenliYang]

Henry, many thanks for your response @HenryLindemann

All the directories are present and access permissions are all fine (arcgisserver, arcgisportal, arcgisdatastore, arcgisimageserverfiles, as well as their subfolders).

There is a folder named with random appended to the characters (e.g., dataFolder_ds_mdlhgrzykfig1fd6) in the Site->Data Store. It cannot be edited or deleted (edit/delete check boxes greyed out).   As I mentioned that the issue occurred shortly after a publishing process during which the publish tool tried to copy data. But the data folder was previously registered and should not be copied (correct boxes checked before hit "Publish"). I tried to cancel the publishing process and it just hung there so I simply closed the Pro. Is it possible that the failed publishing process has anything to do with the problem? 

[HenryLindemann · Accepted solution]

Hi @WenliYang, In 10.8.1 they added new features where a you can add a data source out of portal this is the connection you are seeing in Site -> Data Store. 

the connection is managed out of portal that is why the delete function is grayed out .

On start up is the server loops trough all the services starting them and then stopping them when it is done. I think this process is being interrupted probably by the aborted publishing that causes some sort of problem.

It might also be the .rlock files, I have seen if there crash and they did not get removed the system will not start correctly.

Just make a backup of the system and then stop the server and delete all the .rlock files.

Is your ArcSOC spinning up ? this is the actual services for non-hosted services.

Then you can remove the datastore links -this just removes the link to the datastore it does not remove any data- to just check if the problem is stemming from ArcGIS for Data Store. 

and re-register the connections afterword's

[WenliYang]

Thanks again, Henry @HenryLindemann, for the tips. We were able to remove the datastore items based on your advise. However, the service manager still shows nothing (i.e., not service) although we got rid of all .rlock files and restarted the server ( stop datastore service, stop arcgis service, delete all rlock files in all folders, start service, start datastore service). There was no ArcSOC process present. We encountered got problem of numerous hung ArcSOC processes before but now we got a no service problem. 

[HenryLindemann]

Hi @WenliYang, so the main problem is the ArcSOC's not starting up this is why all the services are missing in ArcGIS Server.

If you have antivirus on the server you can go trough the logs and make sure the antivirus is not blocking ArcGIS paths stuff like pkill.exe for a test you can also stop the antivirus and see if ArcGIS server is starting up.

Something might have corrupted in the back-end so you can  uninstall and then re-install the ArcGIS Server +- 1 hr effort, the uninstall will remove most of the application files but not the content and hopefully also the component that is causing the problem and if you uninstall and re-install to the same directory then you will not have data loss but please make a backup before hand.

On the startup process if you open task manager does the ArcSoc spin up and then spin down ? 

If you set the logging to debug do you see any information in the logs? It might just be that failed service that is crashing out the system.

If you are on maintenance you can also contact you distributor to send out a technician to have a look at the system, it is always good to have a pair of eyes on the issue.

Kind Regards

Henry

[WenliYang]

Hi Henry @HenryLindemann ,

It turns out that we didn't fully get your first tip about the arcgisserver folders. The two subfolders were there and the access permissions were appropriate. But the subfolders under the services were gone with only the files left, which causes the services missing. A customer support engineer helped going through all the steps and checked all the folders/files and found the missing subfolders. 

Many thanks for your time and help,

Wenli

[Kimme]

Hi Everyone,

I'm here for the same reason- our services have disappeared and I cannot seem to bring them back. Our logs, config, and directories are all present and I believe the sub folders have everything necessary in them. 

The configuration succeeds- but I am still directed to the Server Set Up Wizard. 

meaning that has done nothing, I restart services, etc. I am at my wits end here. Any ideas?

Kimme