Error in executing jobs on GeoAnalytics Server and SpatioTemporal big data store ArcGIS 10.5.1

[SuryaSawant]

Q 1. Error in executing jobs on GeoAnalytics Server.

Q 2. Unable to insert data into SpatioTemporal big data store.

To explore capabilities of ArcGIS GeoAnalytics server we have configured following,

1. ArcGIS Enterprise (ArcGIS Server, Portal, web adaptor) - status working

2. ArcGIS SpatioTemporal data store (configured with Portal) 

3. ArcGIS GeoAnalytics Server (federated with portal)

Total Three machines (1 for ArcGIS Enterprise, 1 for SpatioTemporal data store, 1 for ArcGIS GeoAnalytics Server)

Version of ArcGIS - 10.5.1 

Our machine configurations are

OS - Windows server 2012

RAM - 16 GB 

Core - 4

Hard Disk Size - 100GB

Applied patch for ArcGIS GeoAnalytics Server 10.5.1.

Ensured no firewall issues / firewall is switched off.

GeoAnalytics Server Configurations

Maximum allowed compute cores per job (CPU): 2
Maximum allowed memory per job per machine (GB): 4

Refereed documents

Set up ArcGIS GeoAnalytics Server—Portal for ArcGIS (10.5.x) | ArcGIS Enterprise 

Following is the error log of GeoAnalytics Server

Error executing tool. AggregatePoints Job ID: j898c7cb232c645fd9806af7bfd56dec2 : {"messageCode":"BD_101057","message":"Unable to start distributed job. Please check your GeoAnalyticsTools service settings and ensure that there are enough resources available for the job to run."} {"messageCode":"BD_101031","message":"Job has been cancelled."} Failed to execute (AggregatePoints). 

Following is the error log for SpatioTemporal data store

Job 'jaa0d6a1492c64686b2555f8460d6c74d' for tool 'CopyToDataStore' failed: Connect to <configured hostname>:9220 [<configured hostname>/ip addresss] failed: Connection timed out: connect. Connection timed out: connect.

Please suggest on above issues.

[NoahSlocum]

Hello Surya,

In regards to:

""Unable to start distributed job. Please check your GeoAnalyticsTools service settings and ensure that there are enough resources available for the job to run.":

This message is usually seen when GeoAnalytics attempts to use more resources than are available on your machine (see Common GeoAnalytics problems and solutions—Documentation | ArcGIS Enterprise, GeoAnalytics Server settings—Documentation | ArcGIS Enterprise ). However, considering that your GeoAnalytics settings seem to be set appropriately, try checking if another process on your machine is competing with GeoAnalytics Server for resources by observing memory and CPU usage when GeoAnalytics is not running.

The observed message is also often seen when a tool is run while a GeoAnalytics job is already running. Double-check that a GeoAnalytics instance is not already running before you attempt to run a tool by going to the Services tab in Server Manager on your GeoAnalytics machine. In the System folder, you should see the GeoAnalyticsTools service, which will show how many instances of this service are currently in use. This value should be 0 if you are not intentionally running a tool:

If these things do not seem to be an issue, try running a very small job (for example, running the Copy To Data Store tool on a single point feature) to determine if the job given in your example is too large for your current configuration. Finally, if you change your GeoAnalytics settings to 1 GB memory per job per machine and 1 compute core per job, do you still see the error described? 

Once you no longer receive the above message, try running a tool and writing to the Spatiotemporal data store and see if it is successful. If not, try writing to the Relational data store and see if that works. This can help determine more specifically the cause of the data store issue you're seeing.

Also, feel free to contact support if you have further issues, they are available to walk you through any troubleshooting steps over the phone and are often a much faster method of getting help. 

Best regards,
Noah

[SuryaSawant]

Dear Noah,

Thank you for your reply.

The first question (i.e. Q 1. Error in executing jobs on GeoAnalytics Server) is solved for ArcGIS managed GeoDatabase. GeoAnalytics server is able to store results in managed GeoDatabase. The server was restarted and then it stared working. The exact reason for this behaviour is not identified. 

The second question still persists. Following are the server logs,

ArcGIS Server log  

Message: Unable to connect to the manged egdb, Source: Hosted/<feature>.FeatureServer

GeoAnalytics Server log

Message: Error executing tool. CreateBuffers Job ID: j3609b8641b644fada0bfa157c956cecd : {"messageCode":"BD_101030","message":"Failed to execute CreateBuffers. Please contact your administrator with job ID 'j3609b8641b644fada0bfa157c956cecd'.","params":{"toolName":"CreateBuffers","jobID":"j3609b8641b644fada0bfa157c956cecd"}} Failed to execute (CreateBuffers).

Message: Job 'j3609b8641b644fada0bfa157c956cecd' for tool 'CreateBuffers' failed: Connect to <machine fully qualified name>:9220 [<machine fully qualified name>/<IP Address>] failed: Connection timed out: connect. Connection timed out: connect.

SpatioTemporal Big datastore validate response

{
"datastore.name": "bds_2wq79s3h",
"clusterStatus": "YELLOW",
"datastore.overallhealth": "Healthy",
"isHealthy": true,
"numOfNodes": 1,
"numOfPendingTasks": 0,
"activeShards": 1,
"activePrimaryShards": 1,
"relocatingShards": 0,
"initializingShards": 0,
"unassigned_shards": 1,
"validationFailures": [],
"nodes": [{
"name": "<IP Address>",
"heapSize": "7.6gb"
}]
}

Following is the Elasticsearch configuration 

# ---------------------------------- Cluster -----------------------------------
cluster.name: <Cluster Name>
# ------------------------------------ Node ------------------------------------
node.name: <fully qualified server name>
# node.rack: r1
# ----------------------------------- Paths ------------------------------------
path.data: C:/arcgisdatastore/elasticdata
path.logs: C:/arcgisdatastore/logs/<fully qualified server name>/elasticlog
# ----------------------------------- Memory -----------------------------------
# bootstrap.mlockall: true
# ---------------------------------- Network -----------------------------------
network.host: _site_,<fully qualified server name>,_local_
transport.tcp.port: 9320
# --------------------------------- Discovery ----------------------------------
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["<fully qualified server name>:9320"]

Do we need to change discovery.zen.ping.unicast.hosts? As it is allocated to <fully qualified server name> and above response shows the server IP address.

What is the reason for connection timed out (to Spatio Temporal Datastore)?

The firewall on all servers / machines is turned off.

Please suggest regarding the same.

[NoahSlocum]

Hello Surya,

I am glad you were able to fix at least part of your original issue! In regards to the behavior described above, I'm afraid this will require more elaborate troubleshooting than is feasible via GeoNet. Could you please call in to support for further help? If you relay the information above (or direct them to this thread), they will be able to walk you through specific steps you can take to figure out why you're unable to write to your Spatiotemporal Big Data Store. 

Best regards,
Noah

[SuryaSawant]

Hello Noah,

Thank you for your suggestions.

We have resolved the issue. 

It turned out that even after disabling the antivirus on Spatiotemporal Big Datastore Server machine the IP and port of ArcGIS Server and GeoAnalytics server was blocked.

Removing antivirus and allowing ports through Windows firewall resolved the Q2 (i.e. Unable to insert data into SpatioTemporal big data store).

Thanks and Regards,

Suryakant Sawant