How to connect to ElasticSearch - GeoEvent Server/Spatiotemporal Big Data Store

[ZainabAkhtar]

I have the ArcGIS Enterpise (10.9.1) subscription and I would like to expose high volumes of data coming in real-time stored in Elasticsearch through the map/feature services in order to visualize it on a web app. The GeoEvent server is able to handle this huge data ingestion in real-time which can then store it in the Spatiotemporal Big Data Store, as seen in the screenshot attached taken from Esri's DevSummit Event (https://www.youtube.com/watch?v=iW7_w9Evr6c&ab_channel=EsriEvents).

However, I could not find any tutorials on how we can connect to elasticsearch in the GeoEvent server input connectors. I would appreciate any guidance on how i can go about setting up a direct connection to elasticsearch.

[DanWade]

Hello,

Once you have registered the ArcGIS Enterprise within GeoEvent Manager you can create Output layers for adding and or updating features within a big data store. If the GeoEvent Server machine is federated you can just start adding the Output Layers since the ArcGIS Enterprise is already registered.

Spatiotemporal big data stores
https://enterprise.arcgis.com/en/geoevent/latest/administer/managing-big-data-stores.htm

Add a Feature to a Spatiotemporal Big Data Store
https://enterprise.arcgis.com/en/geoevent/latest/disseminate-and-notify/add-a-feature-to-a-spatiotem...

Update a Feature in a Spatiotemporal Big Data Store
https://enterprise.arcgis.com/en/geoevent/latest/disseminate-and-notify/update-a-feature-in-a-spatio...

[JakeSkinner]

@ZainabAkhtar,

After you install ArcGIS Data Store application on a dedicated server, you will have an option to configure a data store.  You will choose "Spatiotemporal" as the data store and then configure this with your ArcGIS Server instance that is federated with Portal.  A common misconception is that you will configure the Spatiotemporal data store with GeoEvent.  That is not the case.

Once the Spatiotemporal Data Store is configured with your federated ArcGIS Server Instance, you will be able to create outputs in GeoEvent that write data to this data store. 

To do this, make sure you have a connection in GeoEvent to your ArcGIS Enterprise instance:

This must be selected for Create Data Source to be active when creating an Big Data Store output:

[RJSunderman]

Hello @ZainabAkhtar --

Adding to what Jake and Dan have said above, the fact that Elasticsearch is the search and analytics engine for Esri's Spatiotemporal Big Data Store is an implementation detail. You should consider the SBDS another type of Enterprise geodatabase, a capability you configure when installing ArcGIS Data Store. Direct connections to the Elasticsearch engine are not supported.

The only supported way to connect and work with data stored in the Spatiotemporal Big Data Store is through tools included with the different ArcGIS Server advanced server roles (e.g. GeoEvent Server, GeoAnalytics, etc.)  Typically client access is limited to the REST interfaces exposed through map/feature services published as you create/publish hosted feature layers.

Your only option for direct connection to Elasticsearch would be to contract with Esri Professional Services for implementation assistance in developing a custom extension to the ArcGIS Enterprise.

Hope this information helps --
RJ

[Anonymous User]

@ZainabAkhtarhave you made any progress with this? I have a customer that is after a similar kind of integration

[JoshPritchard]

Read up on "Custom Data Feeds" associated with ArcGIS Enterprise. Also, there's an app called Koop that could help you out with this while we wait on a release for this feature.