Hosted services not loading with error MapServer not found

[wizgis]

Hello Community,

We are on ArcGIS Enterprise 10.9.1 and today we discovered that all of a sudden our hosted services are not loading. 

Things that we have checked so far:

1. Validated datastore from server manager, datastore was validating
2. Ran describedatastore and it showed datastore mode as READWRITE
3. In developer tool logs error message was : Error Code 400, Service Hosted/<name of service>/MapServer not found 
4. Checked that hosted services are in started state from manager end. 
5. Checked that datastore windows service is running.

Is there anything else that needs to be checked.

Any suggestions on how to resolve this would be really helpful.

[BenClark]

Hoo boy, I used to struggle with this a bunch too--hosted services would run fine for months and months, only to one day inexplicably crash. Turns out any time I needed to restart my Enterprise Windows services on the application server for Portal, Server, and Data Store I hadn't been doing it in the right order, causing the crashes at a later date. This made it a bear to diagnose but I finally figured out that one needs to bring the services back up in a very specific way. If your problem is the same as mine then I recommend following the steps below:

•  Shutdown Order:
1. Portal for ArcGIS
2. ArcGIS Server
3. ArcGIS Data Store
   
   
• Restart Order:
1. Portal for ArcGIS
2. ArcGIS Data Store
3. Wait Ten Minutes, then ArcGIS Server

Now that I follow these steps I've not had hosted services randomly crash for over a year now, and it used to happen every few months. Good luck!

-Ben

[wizgis]

Thanks @BenClark for sharing this with me. Will look into this.

[Matt_Luke]

Followed the above steps and still could not establish a connection to datastore hosted services. 

Restarted the ArcGIS License Manager Service and problem fixed. 

Note that the issue was presenting itself around the time that our Enterprise License was due to rollover into a new Licensing Agreement.

[DavidLight]

We had a problem with random seemingly services stopping a while back and we were also running 10.9.1 Enterprise.  After a lot of head scratching, we found that we needed to update our SQL Enterprise Geodatabase.  After the database update, we haven't had any problems with services stopping or not loading.  This might not be relative to your issue, but possibly worth checking.  Good luck!

[wizgis]

@DavidLight Thank you for sharing your experience with me we are using PostgreSQL Geodatabase. Will look into it.

[JohnLivengood]

Was your ArcGIS Server recently restarted?  If so it might be related to the ArcGIS Server Hosted Services Restart Patch.

https://support.esri.com/en-us/patches-updates/2023/arcgis-server-hosted-services-restart-patch

[wizgis]

@JohnLivengood The only change that has taken place is that our IT team took the backup of the entire machine along with OS and related data and restored on a new machine and since then we are observing this issue on the new machine. So am not sure would this backup and restore would be considered as restart of ArcGIS Server. 

I also tried publishing cities layer (template data) as a hosted feature service with following error message in the logs : Error executing tool. Publish Portal Service Job ID: j25e501827b384a25baa5b0a35f247818 : ERROR 001369: Failed to create the service. Failed to execute (PublishServiceDefinition). ERROR 001369: Failed to create the service. Failed to execute (PublishServiceDefinition). Failed to execute (Publish Portal Service).

[JohnLivengood]

My recommendation is to log into the ArcGIS Server machine and open the Task Manager > Services, look for the ArcGIS Server service and restart it (right click).  If that resolves the issue, installing the patch may help with future outages.