Unable to create hosted feature services - Enterprise 11.1

[Thomas_Puthusserry]

I have a HA enterprise which was upgraded recently   from 10.9 to 11.1. 

2 Portal, 1 GIS server as hosted server, fileshare, 2 datastore and 2 tile cache - these are the VMs involved in the deployment.

Initially there were errors with replication, as visible in the datastore logs from primary datastore VM. I ran validate datastore it returns machine overall health as 'Healthy' for both primary and standby datastore machines.

After this I could create hosted feature services. Now the services are listed under contents, when I click on to see the data, it says there is an error. 

Error: Service Hosted/test5/MapServer not found
Code: 404

Also, I cannot create anymore hosted services. It gives error.

Snippets of logs from GIS Server, Datastore are attached.

Any idea what I need to do ?

Thanks for help

Thomas

[Dan_Brumm]

Tom

 Have you tried to create the service more than once?  I get something similiar when I try and create new Hosted Feature Services. Sometimes they dont publish correctly. It will appear like everything is ok but when I click the data tab I get an error. Typically the fix is to publish the service again. 

Your standby datastore is probably broken. The line:

requestID="">replicationStatus is: NONE</Msg>

means that the Primary and Standby machine are out of sync.  There is no simple fix for this that I have found. When I get this I typically remove the Standby Datastore and re-add it. 

[Thomas_Puthusserry]

Thanks @Dan_Brumm . To fix the replication issue I am following this step:

https://support.esri.com/en-us/knowledge-base/error-arcgis-data-store-has-detected-a-replication-fai...

However when I tried to remove the standby machine got error and couldn't progress further. 

removemachine <machine name> --store {relational | tileCache | spatiotemporal} [--force {true | false}] [--prompt {yes | no}]

 Is there a clear step by step process documented anywhere in relation to fixing replication issues in HA datastore setup?

Thomas 

[Dan_Brumm]

Hey Tom

The fun never stops with Enterprise software 🙂 

What did the error message say?  I have a couple off the cuff suggestions..

1. You have to run the RemoveMachine command from the Primary Datastore  Machine

2. You have to run the Command Prompt as an administratrator

3. The syntax should look like:

                    removemachine <standbyDSHost> --store relational --force true --prompt yes

                  Don't include the <> with the standbyDSHost name

[Dan_Brumm]

Also, there is no documented instructions that I could find on fixing the replication issue. Once is breaks you really have no choice but to remove and re-add. 

[Thomas_Puthusserry]

I have now removed the standby datastore and added again. One warning in the ArcGIS server log is not clear: 

WARNING

Error saving the log settings to the ArcGIS Data Store configuration store. null

Full server log relevant to this is attached. Any idea?

@HenryLindemann can you suggest / help? 

Thanks Thomas

[hlindemann]

Hi @Thomas_Puthusserry , can you do a describe on the datastore and see what that says. usually, it is under c:\progam files\arcgis\datastore\tools

if there is a problem with the sync it might indicate that the wrong server is the primary. how large is the datastore ?

can you also double check that the ports are open bi - directionally?  

[Thomas_Puthusserry]

Hi @HenryLindemann I have raised this as support ticket with ESRI UK (caseNumber=03528122) and attached all log files with that ticket.

Basically there are no errors highlighted in the describedatastore tool. The validate datastore indicates the primary is primary and secondary is standby.

There is no t any hosted data here, this is our test env. All ports are allowed internally.

Thanks for helps. Thomas

[Dan_Brumm]

I would double check permissions.. Make sure the Datastore service account has full access to the datastore directories.  Also keep an eye on the logs, that error message may just clear up on its own.