Help with intermittent(WARNING/SEVERE) logs on ArcGIS server manager.

346
7
Jump to solution
04-22-2024 01:17 AM
Yogesh_Chavan
New Contributor III

We have a multimachine ArcGIS Enterprise deployment and since last week we are seeing the below waning and SEVERE logs on ArcGIS Server Manager.

All the services seems to work fine, datastore machine is working fine as well, and there is no communication conflict between machines hosting server, portal and datastores.

Yogesh_Chavan_1-1713773622735.png

Did someone encountered such logs before? What could be triggering this?

0 Kudos
1 Solution

Accepted Solutions
A_Wyn_Jones
Esri Contributor

@Yogesh_Chavan it's likely that the Server to Datastore machine communication is blocked by port configuration. 

Have a look at this port diagram https://enterprise.arcgis.com/en/system-requirements/latest/windows/pdf/ports-enterprise-deploy-dgm....

Note that port 2443 is required both inbound/outbound between Server and Datastore - loading the web page https://Datastoremachine.domain:2443/arcgis/datastore from the Server machine is a good way to test this.

When registering Datastore, I'd suggest using ArcGISServermachine.domain to register with ArcGIS Server machine - you shouldn't need to configure through the load balanced endpoints unless there's a specific configuration requirement e.g. highly available stack.

To check that Datastore can communicate with your Server - web browse to https://ArcGISServermachine.domain:6443/arcgis on the Datastore machine.

 

"We've boosted the Anti-Mass Spectrometer to 105 percent. Bit of a gamble, but we need the extra resolution."

View solution in original post

7 Replies
A_Wyn_Jones
Esri Contributor

Hi @Yogesh_Chavan,

It looks like there may be a permissions error on ArcGIS Datastore which is preventing ArcGIS Server to read the logs of Datastore.

"We've boosted the Anti-Mass Spectrometer to 105 percent. Bit of a gamble, but we need the extra resolution."
Yogesh_Chavan
New Contributor III

Hello @A_Wyn_Jones,

Thank you for sharing your insights.

Kindly find the response below:

  • Are all components running from the same ArcGIS Service account?

Yes

  • Can you please verify the permissions of c:\arcgisdatastore 

Service account have full control over this

Yogesh_Chavan_2-1713785562712.png

 

Yogesh_Chavan_1-1713785507231.png

Not sure what is going wrong. We actually started observing this since last 2-3 weeks. 

0 Kudos
A_Wyn_Jones
Esri Contributor

@Yogesh_Chavan It looks like your Datastore is offline.

On the datastore machine, can you visit https://localhost:2443/arcgis/datastore and see a webpage?

It's also worth checking that you can get to https://Datastoremachine.domain:2443/arcgis/datastore from the ArcGIS Server machine - this will check if any networking has changed.

 

This issue would only be noticeable if you're viewing Hosted content in your Portal - if the majority of your layers are hosted on a separate Database like SQL or Postgres then you wouldn't necessarily notice. 

"We've boosted the Anti-Mass Spectrometer to 105 percent. Bit of a gamble, but we need the extra resolution."
Yogesh_Chavan
New Contributor III

Thank you @A_Wyn_Jones for you helpful insights.

To our findings we tried following your suggestions and observed that the server machine fails to connect to https://Datastoremachine.domain:2443/arcgis/datastore.

Yogesh_Chavan_0-1713872111311.png

This works well in datastore machine for both localhost and domain URL.

Yogesh_Chavan_1-1713872282709.png

There is one additional info that I think is quite suspicious. We are using NLB URLs configured on top of our web adaptor URLs for accessing server and portal, and those were configured before installing and registering datastore machines with the server.

Can this could be a possible trigger for this issue? Or should we focus on why this domain URL fails to communicate from the server machine?

Adding some more info from the describedatastore.bat utility we observed that the machine with relational datastore's owning system URL is web adaptor URL whereas the other machine with tile cache datastore shows Port URL.

We have 2 datastores installed on separate machines, and the issue with errors/logs is in relational.

Yogesh_Chavan_0-1713885500289.png

 

0 Kudos
Yogesh_Chavan
New Contributor III

Hello @A_Wyn_Jones,

Were you able to review the above details? Just in case I missed out on some critical info earlier.

Also, would it be a good idea to un-register the relational datastore from the server and re-register using the Port or NLB URL and not the Web Adaptor URL?

Reading through the docs makes us clear that hosted services would not work? Does a re-registered datastore makes the services operate normally?

Thank you so much. Feel free to review when you have some off hours:)

0 Kudos
A_Wyn_Jones
Esri Contributor

@Yogesh_Chavan it's likely that the Server to Datastore machine communication is blocked by port configuration. 

Have a look at this port diagram https://enterprise.arcgis.com/en/system-requirements/latest/windows/pdf/ports-enterprise-deploy-dgm....

Note that port 2443 is required both inbound/outbound between Server and Datastore - loading the web page https://Datastoremachine.domain:2443/arcgis/datastore from the Server machine is a good way to test this.

When registering Datastore, I'd suggest using ArcGISServermachine.domain to register with ArcGIS Server machine - you shouldn't need to configure through the load balanced endpoints unless there's a specific configuration requirement e.g. highly available stack.

To check that Datastore can communicate with your Server - web browse to https://ArcGISServermachine.domain:6443/arcgis on the Datastore machine.

 

"We've boosted the Anti-Mass Spectrometer to 105 percent. Bit of a gamble, but we need the extra resolution."
Yogesh_Chavan
New Contributor III

Thank you @A_Wyn_Jones for your insights.

0 Kudos