Latest Contributions by RJSunderman

The GeoEvent product team would like to announce the release of a new tutorial introducing the Spatiotemporal Big Data Store. The tutorial is available on the GeoEvent product gallery with our other product tutorials. Here is a direct link to the new tutorial:  Tutorial - Spatiotemporal Big Data store Feedback is welcome. Please add any comments you have to the item in the gallery or send an e-mail to geoevent@esri.com Best Regards -- RJ ... View more

The GeoEvent product team would like to announce the release of a new tutorial introducing the Spatiotemporal Big Data Store. The tutorial is available on the GeoEvent product gallery with our other product tutorials. Here is a direct link to the new tutorial:  Tutorial - Spatiotemporal Big Data store Feedback is welcome. Please add any comments you have to the item in the gallery or send an e-mail to geoevent@esri.com Best Regards -- RJ ... View more

I wanted to take just a moment to follow-up some questions posed by staff attending the inaugural boot camp introducing the new 10.4 Spatiotemporal Big Data Store. The boot camps are not available for delivery as customer training. They are focused opportunities for Esri staff to quickly ramp-up on products so that they can engage with customers, provide advice, and transfer knowledge. If you are reading this as a customer working with Esri staff, mention the boot camps - not everyone knows that they are available. What are the plans, if any, to provide direct connections to Elasticsearch in order to access the underlying data? When will we be able to batch load data into a BDS? Offload/archive data from a BDS to a system off-site? Monitor the activity and what data is actually in each node of a 5-node BDS data ring? There are no current plans to provide customers direct access to the Elasticsearch search server or the underlying Lucene library functions. The BDS is a non-sql database provided as a managed database for archiving and accumulating large volumes of real-time observations, similar to how a PostGRE relational database is provided as a managed database for storing feature data. It is possible to obtain the connection information and connect to the PostGRE RDBMS through an application such as ArcCatalog - but this will not be supported by Esri Technical Support. If your organization already has a high volume of data stored in an Elasticsearch/Lucene solution and you would like help exposing that data through the map services and feature services normally created when creating a new data source in Esri's BDS, it is possible that Esri Professional Services can help you achieve this as a contracted project. The product team anticipates that users will be able to batch load data into an Esri BDS as part of the next major product release (e.g. "10.5") using new functionality being created for GeoAnalytics. We are working now on a solution which will allow data to be offloaded for archival - this should be available as part of the GeoEvent product later this year. There are currently no plans from the GeoEvent product team to design a database monitoring utility for the spatiotemporal big data store. Database administrators familiar with tools such as Apache Ambari or the Splunk App for HadoopOps are asked to work with their Esri technical advisors or account representatives to define product requirements and functionality needed in a "BDS Administration and Monitoring" application which can be communicated to Esri's ArcGIS Data Store product development team. How is the system memory configuration for the BDS exposed?  Is this configurable? Given a project which can only deploy a single server machine with 32GB of RAM... It would be nice if we could allocate 6GB to the GeoEvent JVM and no more than 8GB to the BDS process(es) It should be possible to configure memory allocation to the BDS. Shengyao Duan from the ArcGIS Data Store product team will have more information.  The product's default (and Esri's recommendation) is that 50% of a server's RAM be allocated to the BDS (up to 32GB). Because the BDS is so memory intensive it is recommended that you install the ArcGIS Data Store and enable the BDS on a dedicated server. Details on a recommended systems architecture for leveraging the spatiotemporal big data store are laid out in a tutorial introducing the product, due to be released very soon. The product team's current recommendation is that a "WebGIS" be built out on one server machine (with Portal, ArcGIS Server as a hosting server, and a relational database as the hosting server's managed geodatabase). A separate "RealTime" server (with a second ArcGIS Server and the GeoEvent Extension) would provide dedicated system resources for real-time event processing, and a third server machine would host the BDS. [ June 22, 2016 ] Randall Whitman - Notes on configuring BDS RAM usage: arcgis/datastore/tools/changedbproperties.bat --store spatiotemporal --heap-size 4096 Say you have a large system of stationary sensors. Since the sensors themselves are not moving, you don’t need to send a Geometry with every event. Could we offer a “Related Features” functionality for BDS like we have for Stream Services? GeoEvent treats Geometry like any other attribute field. So a GeoEvent might have several different fields of type Double, Date, String ... and Geometry.  The product team is considering an enhancement which would allow non-spatial tables to be supported in a BDS.  However, at the initial release, event records processed through GeoEvent are expected to have a Geometry. The "Related Features" capability provided for stream services is not part of a standard map service or feature service. It's really just a URL provided by the stream service to clients who can use the URL to discover a feature service and perform a join with the non-spatial event in order to "enrich" the event with geometry from features in an external service. This work is all performed within a stream layer - so no - it is unlikely that we will be able to offer a "Related Features" capability for data being stored in a BDS. If every sensor (all 1.7 Million) were sending not just their current reading, but a forecast for each of the next 150 hours, so every two hours you were getting 150 readings from each sensor, any ideas on how we could handle and potentially visualize the inbound data? The GeoAnalytics functionality being developed will provide different ways of aggregating data records retrieved from a BDS. It should be possible to perform statistical aggregation (minimum, maximum, average, etc.) based on an attribute field whereas the initial map service released with the 10.4 product only aggregates based on event count in a grid cell / area. Visualization of multidimensional data, such as an array of predictive values from a single sensor, is intriguing. The GeoEvent team would be interested in discussing how such data would be collected, processed, stored, and visualized as part of next-generation Real-Time GIS development currently underway. Any idea if we’ll be able to enable the BDS using the web form post-install at 10.4.1 rather than dropping down to use a command-line invocation of an administrative batch script? Enablement of a spatiotemporal big data store should be included on web forms provided by the ArcGIS Data Store product for the next software release (e.g. "10.4.1").  The only reason this was not integrated for 10.4 was an internal freeze on user interfaces which occurred before design and development had been completed for the BDS. Please confirm any expected changes to the ArcGIS Data Store with Shengyao Duan​. Hope this information is helpful -- RJ ... View more

Changes made to the Text adaptor for the 10.4 product release might catch a few folks off guard, so I want to call attention to what you might expect to see and why. Several inbound connectors – specifically those using the Generic-JSON and XML adaptors – provide the ability to ‘Construct Geometry From Fields’ so that when event data is received which contains coordinate information on a point location a Geometry can be constructed by the GeoEvent input prior to sending the event to a GeoEvent Service. At the 10.3.1 release, inputs such as ‘Receive Text from a TCP Socket’ and ‘Watch a Folder for New CSV Files‘ do not have this capability because it was not included in the Text adapter. It was observed that if a user allowed the input to create a GeoEvent Definition and then later reconfigured the input to request that it construct a Geometry from specified event fields without also editing the event definition to include a Geometry field in which the constructed Geometry could be placed … that the input would stop ingesting event data. For the 10.4 release, an enhancement was made to the Text adaptor to bring it in line with the Generic-JSON and XML adaptors: Issue #945:  Text adapter needs to follow Generic-JSON adapter's behavior and create a field of type Geometry, tagged GEOMETRY, when creating a GeoEvent Definition Now, at the 10.4 release, when you select the ‘Construct Geometry From Fields’ option and specify the name of event attribute fields containing coordinate values, the GeoEvent Definition generated for you by the input will include a field named “Geometry”.  The field will be added as the last field in the event definition and will be tagged with the GEOMETRY GeoTag. If you leave the ‘Construct Geometry From Fields’ parameter set to its default ‘No’ the generated event definition will not have the Geometry field highlighted above. If you elect to edit the generated GeoEvent Definition to replace ‘Field1’, ‘Field2’, ‘Field3’ (etc.) with meaningful names, but neglect to include a field in the GeoEvent Definition into which the constructed Geometry can be placed, and then alter your input to switch ‘Construct Geometry From Fields’ from ‘No’ to ‘Yes’ … beginning with the 10.4 release the input will alter the event definition for you to create the needed Geometry field. Here’s where this enhancement might produce undesired behavior … Let’s say that you had been using GeoEvent at the 10.3.1 release with a ‘Receive Text from a TCP Socket’ input successfully receiving event data sent from the GeoEvent Simulator. Your GeoEvent Definition might look something like the following: You export your 10.3.1 GeoEvent product configuration as an XML file, upgrade to 10.4, and import your configuration. When you try simulating the same data which was working at the 10.3.1 release, the input’s event count no longer increments and it does not appear that your GeoEvent is receiving any of the data you’re simulating. You look in the GeoEvent logs and see a message: Cannot find matching GeoEvent Definition. Event is not created. The incoming text is SWA2706,3/16/2012 02:25:30 PM,IAD,TPA,B733,37000,-79.585739,34.265521 What’s going on? It is likely that the field named “Location” is not the field the ‘Construct Geometry From Fields’ operation expects. Even though you were careful to copy and edit the GeoEvent Definition, back in 10.3.1, to include a field whose data type is Geometry and tag the field GEOMETRY … its name is not “Geometry”, so the input doesn’t recognize it. Remember that delimited text is not well-defined like JSON or XML. If the inbound events contained field names then the adapter would be able to map the received data into the correct fields in the event definition. The 10.4 Text adapter does not assume that a field of type Geometry is sufficient (even the field is tagged GEOMETRY). It looks for a field named “Geometry” and, failing to find one, it logs the error shown above and discards the event. To fix the problem, all you have to do is edit the GeoEvent Definition imported with your configuration and rename the last field from “Location” to “Geometry”. This is probably a scenario that many users will encounter when upgrading 10.3.1 configurations which include a TCP/Text input to the 10.4 release. You need to remember that at 10.4 your Geometry fields should be named “Geometry” (not “Location”, or “Target”, or “Geom”). The field must still be the last field in the GeoEvent Definition. Hope this information saves you a little future grief - RJ ... View more

Technical Support is available on this topic. Please contact Esri Technical Support and mention Article ID 000011685 and Article ID 000011689 See also:  Geoevent WebSockets in 10.6 with ARR‌ and Windows/IIS Reverse Proxy for GeoEvent WebSocket  I received a second request recently for the information below, provided to me originally by Javier on the GeoEvent Server development team, so I thought I’d post it in case others are looking for the information. After installing ARR, double-click the URL Rewrite icon and create a similar rule (replacing the hostname of the Rewrite URL to match your environment):   After applying the rule confirm you are able to access GeoEvent Manager through IIS:   Configuring your SSL certificates will be important. You can use IIS to create and export a Domain Certificate with a private key and then configure ArcGIS Server and GeoEvent to use that certificate. An alternative would be to get IIS to trust the certificate that is configured with your ArcGIS Server/GeoEvent installation.   It was discovered that, using the 10.3.1 release of GeoEvent Server and IIS version 8.5 with WebSockets enabled, with ARR configured to create a reverse proxy for GeoEvent that StreamService connections did not work. This issue was addressed in the 10.4 release of GeoEvent Server.   10.3.1 deployments can create a reverse proxy similar to WebAdaptor specifically for WebSockets using NGINX.  Steps outlined below provide some detail on installing and configuring NGINX. A sample nginx.conf configuration is attached which you can use as a reference.   Please note:  Esri tech support cannot help troubleshoot reverse proxy configurations - whether they be NGINX, IIS, or Apache.  If you do plan on deploying a reverse proxy as part of a systems solution, make sure you have someone with appropriate experience available to you for help troubleshooting. Esri tech support has a KB article you can refer to: http://support.esri.com/cn/knowledgebase/techarticles/detail/40680   Install a WebSocket reverse proxy server Nginx. Enable HTTPS on the proxy server and configure it to use a certificate issued by a trusted 3rd party CA (Thawte, VeriSign, DigiCert). Configure the proxy server to forward requests to GeoEvent Services. Attached is an example of an Nginx configuration file. You will need to change the following settings: server_name <your Web Socket reverse proxy server name> ssl_certificate <your CA-cert certificate file> ssl_certificate_key <your CA-cert certificate key file> server <list of servers that GeoEvent Extension is running on>. In this example, the server names are dev01014.esri.com, dev01015.esri.com, and dev01018.esri.com   Note: The WebSocket reverse proxy server is set to use HTTPS, but it connects to HTTP on the backend. In the example configuration file, the proxy_pass for port 443 is set to http: //skivmHTTP; not https: //skivmHTTP.   In a web browser, navigate to http: //<ServerName>:6080/arcgis/admin to access the ArcGIS Server Administrator Directory. Login and click system -> properties-> update. Enter a property called WebSocketContextURL to point to the Web Socket reverse proxy.   For example: {"WebSocketContextURL": "wss://skivm.esri.com"}   In a browser, navigate to: https: //<WAMachineName>/<context>/rest/services/<StreamServiceName>/Streamserver/subscribe   Click Subscribe to verify data is streaming. You can also verify data is streaming by navigating to: https: //<WAMachineName>/<context>/rest/services/<StreamServicename>/StreamServer?f=jsapi   See Also: https://nginx.com/resources/wiki http://nginx.com/resources/admin-guide/reverse-proxy http://www.iis.net/downloads/microsoft/application-request-routing http://www.iis.net/learn/extensions/configuring-application-request-routing-(arr)     Hope this information is helpful - RJ ... View more