topic Re: Temporarily missing Data Stores on one node in ArcGIS Enterprise Questions

Temporarily missing Data Stores on one node

Peter_von_Minden — Thu, 03 Apr 2014 21:25:01 GMT

SETUP:

Two instances of ArcGIS Server 10.1 installed on two separate machines that are clustered.

ISSUE:

We have been experiencing a strange occurrence where, on one of our nodes, the Data Stores appear to be missing (both Registered Databases and Folders) in the ArcGIS Server Manager for that node and the other node shows them. If we restart ArcGIS Server on that node, we still come up with an empty list of Data Stores. It has no ill effect on our services but it does stop us from publishing services to that node as they do not exist. However, if I try to re-add them, I get a message telling me that they already exist. I get errors like this in the Logs:

ERROR MESSAGE:

SEVERE-Apr 4, 2014 9:03:25 AM-Unable to find the data source 'com.esri.arcgis.discovery.admin.dataspace.DataItemFilter@36e5713e'.-Admin

SELF HEALING:

The strangest thing about it is that after a few days of it not showing up, it suddenly appears.

ANY HELP OUT THERE:

Has anyone else experienced this before and what did you do to fix it other than waiting until it fixes itself?

Re: Temporarily missing Data Stores on one node

WilliamCraft — Thu, 03 Apr 2014 23:46:22 GMT

Where are the 'arcgisserver' directories located for the entire ArcGIS for Server site? Are they shared correctly to both machines? Take a look at the arcgis-dataspace.xml file within ...\arcgisserver\config-store\data folder directory to verify that the data store information is present. Validate your data stores by going to https://<FQDN_host>/arcgis/admin/data/validateAllDataItems, selecting JSON, and clicking the Validate button. Do you get a 'success' message?

Some other things to check... browse to https://<FQDN_host>/arcgis/admin/clusters/default and verify that the configured state of your Cluster is "STARTED". Then, click the Machines link under your Cluster section to verify that both your machines are still listed. Also, go to https://<FQDN_host>/arcgis/admin/machines to verify that both machines still appear in your site.

Re: Temporarily missing Data Stores on one node

Peter_von_Minden — Fri, 04 Apr 2014 04:20:42 GMT

On both machines, the arcgis-dataspace.xml file appears to be identical and has the data stores that I am expecting.

After going to https://<FQDN_host>/arcgis/admin/data/validateAllDataItems, selecting JSON, and clicking the Validate button, on the working machine I get the following message:

"status": "error",
"messages": ["JSONObject[\"dataItems\"] not found."]

and the dodgy one (no data stores showing up) I get:

"status": "error",
"messages": ["Error loading the dataspace from file."]

Browsing to https://<FQDN_host>/arcgis/admin/clusters/default I can verify that the configured state of the Cluster is "STARTED" on both sites. Same with the Machines link under your Cluster section as both machines are listed. I also went to https://<FQDN_host>/arcgis/admin/machines and verified that both machines still appear on the site.

Re: Temporarily missing Data Stores on one node

WilliamCraft — Fri, 04 Apr 2014 10:05:14 GMT

I believe that in a cluster the config-store needs to exist in a shared location accessible to all site nodes. What you're describing almost sounds like you have two separate clusters within one site or two separate sites altogether. I can't tell for sure though.

In any case, make sure that both site nodes have access to the SDE connection files and to the folders that you registered as data stores. Are you using a domain account, a local account, or the local services account to run the ArcGIS for Server Windows service on these machines? That account should have read/write access to your registered data stores.

Re: Temporarily missing Data Stores on one node

Peter_von_Minden — Sun, 06 Apr 2014 23:11:05 GMT

We have two separate sites with a Gluster File share system set up for automatic replication. So whatever we do on one site will be replicated to other site in real time. The sites are Linux machines by the way.

I can connect to the SDE databases on both sites.

We are using a local account to run the ArcGIS for Server Windows service and the account user does have read/write access.

I am going to organise a Gluster restart today to see if that resolves it, failing that, a machine reboot and will post the results.