Hosted Services not working after configuration changes to Server

[RobMcCulley]

We've been running a single tier Enterprise deployment, and decided to expand it to a two tier deployment to improve performance.  I'm following the instructions here to migrate Server and Data Store to a new machine.

Step 1 of migrating a machine running Server is to move the server directories and configuration store to a shared network location.  After performing this step, all of our hosted feature classes no longer work.  In Server Manager, the hosted feature classes are shown as "Stopped":

I've tried restarting Portal, Server, Data Store, and the original Single Tier machine.  None of those worked.

I can't see anything wrong with the Data Store.  The output of describedatastore.bat:

General information of ArcGIS data stores on FQDN1
==================================================
Data store release..................11.1.0.42869
Staging location....................C:\arcgisdatastore\staging
Log location........................C:\arcgisdatastore\logs
Free disk space.....................30.00GB
Threshold for READONLY mode.........1024MB

Information for relational data store ds_oaflnz7l
==================================================
Backup location.....................C:\arcgisdatastore\backup\relational
Is backup folder shared.............false
Backup schedule.....................{"schedule-starttime":"00:00:00","schedule-frequency":"Every 0 DAYS"}
Days backup retained................7
Data store status...................Started
SSL enabled.........................true
member machines.....................FQDN1
Maximum connections.................150
Owning system URL...................https://FQDN1:6443/arcgis/admin
Portal for ArcGIS URL...............https://webadapter/portal
Number of connections...............0 connection(s) to managed database
Data Store mode.....................READWRITE
Is Point-in-time recovery enabled...No
Query optimizer enabled.............Yes




Operation completed successfully.
PS C:\Program Files\ArcGIS\DataStore\tools>

The output of validating the DataStore from the Server Admin interface:

{
  "datastore.release": "11.1.0.42869",
  "datastore.name": "ds_oaflnz7l",
  "datastore.replmethod": "ASYNC",
  "datastore.isReadOnly": "false",
  "datastore.isConfigured": "true",
  "machines": [{
    "machine.overallhealth": "Healthy",
    "datastore.release": "11.1.0.42869",
    "datastore.release.configstore": "1.5",
    "platform": "Windows",
    "machine.isReachable": "true",
    "hostip": "XXX.XXX.XXX.XXX",
    "name": "FQDN1",
    "role": "PRIMARY",
    "dbport": 9876,
    "initstarttime": 1695231440090,
    "healthcheck.enable": "true",
    "status": "Started",
    "adminurl": "https://FQDN1:2443/arcgis/datastoreadmin/",
    "db.isactive": "true",
    "db.isAccepting": "true",
    "db.isInRecovery": "false",
    "db.ActiveReplMethod": "NONE",
    "db.isManagedUserConnValid": "true",
    "datastore.release.pg": "14.5",
    "datastore.release.sde": "11.1.0",
    "datastore.release.geometry": "1.30.4.10",
    "datastore.release.geometrylib": "1.30.3.10",
    "db.isSiteConnValid": "true"
  }],
  "datastore.release.configstore": "1.5",
  "datastore.release.geometry": "1.30.4.10",
  "datastore.release.geometrylib": "1.30.3.10",
  "datastore.release.sde": "11.1.0",
  "datastore.release.pg": "14.5",
  "datastore.layer.extent.updated": false,
  "datastore.status": "Started",
  "datastore.isActiveHA": "false",
  "datastore.overallhealth": "Healthy",
  "datastore.lastfailover": -1,
  "datastore.lastbackup": 1684955920681,
  "datastore.isRegistered": "true",
  "datastore.hasValidServerConnection": "true",
  "datastore.validServerMachinesList": [
    {
      "machineName": "FQDN1",
      "adminURL": "https://FQDN1:6443/arcgis/admin"
    },
    {
      "machineName": "FQDN2",
      "adminURL": "https://FQDN2:6443/arcgis/admin"
    }
  ],
  "owningSystemUrl": "https://FQDN1:6443/arcgis/admin",
  "status": "success"
}

If I connect directly to the DataStore database using Pro, all of my feature classes are there, and if I add them to a map, the data is still there as well:

Any idea what I need to do to get these services working in Portal/Server again?

[DavidColey]

Have you tried using the python api to start the services?  From time to time, some of my hosted feature services or vector tile services will 'stop'  I restart them using the python api.  Something like:

try:
    gis = GIS("https://yourdomain.net/portal", "your_portal_login", "your_portal_pw")

    gis_servers = gis.admin.servers.list()
    print(gis_servers)
    #message = message + "\n" + str(gis_servers)

    agshost = gis_servers[0]
    print(agshost)
    #message = message + "\n" + str(agshost)

    vtServicesList = agshost.services.list(folder='Hosted')
    for vtService in vtServicesList:
        serviceName = vtService.properties.serviceName
        if (serviceName == 'ExoticInvasiveSpeciesPoint' or serviceName == 'ExoticInvasiveTreatment'):
            print(serviceName)
            message = message + "\n" + str(serviceName)
            vtService.start()
            print(vtService.status)
            message = "\n" + message + "\n" + str(vtService.status)
    
except Exception:
    # If an error occurred, print line number and error message
    import traceback, sys
    tb = sys.exc_info()[2]
    e = sys.exc_info()[1]
    #print(e.args[0])
    #print "Line %i" % tb.tb_lineno
    message = message + "\n" + "Line %i" % tb.tb_lineno
    message = message + "\n" + str(e)

works for me every time I encounter a stopped hosted service.

Hope this helps