Latest Contributions by RJSunderman

Hello Felix - The AIS data stream you illustrate looks like raw 7-bit encoded NMEA-0183HS AIS data. The GeoEvent Extension does not provide an adapter which is capable of decoding this format. NMEA-0183HS messages are a variation of the NMEA 0183 data standard. They are 7-bit encoded ASCII messages which are not human readable. While they are part of the broader NMEA specification, they are not what common consumer devices like Garmin or smart phone applications broadcast. The NMEA messages the GeoEvent Extension is able to ingest are 8-bit NMEA-0183 messages which typically look more like the following: The full NMEA specification details a variety of different message sentence structures. There was a pull-request, for example, on the Esri / nmea-for-geoevent GitHub project to modify the sample connector we have on the gallery to support RMC sentences of different lengths. Having open-sourced our NMEA implementation, developers are able to tailor the connector to match their device's sentence structure - if what they are receiving does not happen to match the GGA / GSA / GSV / RMC sentence structures illustrated above. Since the NMEA-0183HS AIS data you are receiving cannot be adapted by the NMEA connector on the Gallery, have you considered the exactEarth Latest Vessel Information connector available from the Partner Gallery? Best Regards - RJ ... View more

The GeoEvent extension includes an inbound connector which is able to poll an existing Esri feature service, for example, to retrieve data from a feature layer. The feature service in this case acts as a data broker which the GeoEvent extension can work through to access the underlying feature data in the database. The product team has no plans to provide ODBC / JDBC connectors to retrieve data directly from RDBMS tables. Connectors such as these would be database specific and we are working to keep GeoEvent database agnostic. You do have a few options available, however, without relying on an ODBC / JDBC connector: Since ArcMap will not allow you to publish a map document as a feature service if the MXD has no feature layers, you cannot create an MXD with *only* a non-spatial table and publish a feature service. ArcMap will require that you include at least one feature layer whose "features" come from a "feature class" … a spatial table containing geometry. However, if you add your non-spatial table to a map document and then add any feature layer, even one pointing to an empty (dummy) feature class which has no features, you should be able to publish a feature service. An out-of-the-box GeoEvent input can then poll the non-spatial table for "features" through the feature service. When configuring the Poll an ArcGIS Server for Featuresinbound connector, you simply select the non-spatial table as the "layer" rather than the actual (dummy) feature layer which will be forever empty (no features). Another option is to create a script which would periodically retrieve the non-spatial event data from the database's table, and then HTTP/POST the data directly to an endpoint associated with a GeoEvent Extension input. The out-of-the-box inbound connectors expect either Generic JSON or Esri Feature JSON. Later releases of GeoEvent also support XML and geoJSON. You might also look into creating a wrapper around the RDBMS using an open API such as the OData protocol (http://www.odata.org/). The OData protocol provides rapid solution development incorporating database access through web services as a front facet to the backend database. We are only interested in retrieving row data as "events" … and the OData protocol provides an REST endpoint through which the database records can be retrieved as generic JSON. This approach fits well with the GeoEvent extension which provides a generic JSON adapter out-of-the-box. You can then configure a Poll an external website for JSON inbound connector to poll the OData endpoint for data – or configure a Receive JSON on a REST endpoint inbound connector to receive JSON data from an OData HTTP POST (though I'm not sure if OData is capable of retrieving the database records and POSTing them to an external client). An architecture for the OData approach might look something like the following: Hope this information is helpful - RJ ... View more

Andreas - Some feedback specifically on your question:  "Why is the GEP trying to access an admin URL?" https://<Web adaptor URL>/arcgistest/admin/data/findItems?f=json&token=XXXXXXX&types=egdb: GET Request failed(HTTP/1.1 403 Forbidden). The GeoEvent Extension is using the admin API to determine if a managed geodatabase has been registered with ArcGIS Server. If there is a managed geodatabase, when configuring a new 'Send Features to a Stream Service' output and selecting to publish a Stream Service, you will have the option to check the 'Store Latest' checkbox and publish a "buddy" feature service which will cache the most recent observation for each received TRACK_ID. If you do not have a managed geodatabase registered with ArcGIS Server, the 'Store Latest' option is not available. - RJ ... View more

Hello Andreas - My apologies that no one has gotten back with you in response to your question. GeoEvent was designed to rely on GIS-tier communications with an ArcGIS Server site using Server generated tokens. Support for web-tier authentication and authorization using SAML and IWA is being developed, but is limited, and is being introduced beginning with the 10.3.1 product release. I need to clarify your question. Are we working with two different GIS Server machines, both with ArcGIS Server installed, which are participating in different ArcGIS Server sites? I ask because you mentioned the term "cluster" and I want to make sure that the GIS Servers we are considering are not part of a single ArcGIS Server site cluster. If the two GIS Servers were participating in the same ArcGIS Server site cluster with the GeoEvent Extension installed on one GIS Server but not on the other we would be working with an unsupported configuration. All of the GIS Servers in a site's cluster must be provisioned exactly the same - this is important when using the GeoEvent Extension. Next question - is it an option for your first ArcGIS Server, the one which has the GeoEvent Extension installed, to reach the second ArcGIS Server without going through the Web Adapter? Design assumptions built-in to the GeoEvent 10.3 release (and previous releases) assume that GeoEvent will be able to discover services using the ArcGIS REST Services Directory and that the directory will be accessed through either the open REST endpoint http://server‑name.domain:6080/arcgis or the secure endpoint https://server‑name.domain:6443/arcgis. The preference would be that GeoEvent not attempt to go through the web adapter to discover a server's services. Some support for IWA was implemented in the 10.3.1 product release for registering ArcGIS Server connections as GeoEvent Data Stores. For example, you can specify that GeoEvent should use a token to access the data store and specify the secure endpoint (https://server‑name.domain:6443/arcgis) as the URI to be used. When the token expires, you will have to refresh your Data Store configuration with a new token in order to continue using the data store. You can also, beginning with the 10.3.1 release, select to use web-tier authentication and supply a recognized IWA username and password when registering a data store. My understanding is that, even when using web-tier authentication, that access must still be made through fully-qualified server-name / port URI ... either the http://server‑name.domain:6080/arcgis endpoint or the secure endpoint https://server‑name.domain:6443/arcgis. I will try to get someone to confirm this. Hope this information helps - RJ ... View more

Hello Felipe - Have you had a chance to look through the exercises in the Notifications in GeoEvent tutorial? The reason you see the event count increasing on your email-text-out output is that the output has successfully taken a GeoEvent from an internal queue, adapted the event data to produce a plain/text message, and sent that message to the specified SMTP Server as a request to generate an e-mail message to the specified recipient(s). This is a “fire-and-forget” type of operation. The GeoEvent Extension will not contact the specified SMTP server to determine its status, and there is no way for GeoEvent to process acknowledgement or error messages which might be sent from the SMTP server. You cannot assume that because an output incremented its event count that an email has been successfully generated, delivered, or received. The incremented event count only indicates that a message has been sent to the specified SMTP server for delivery to the specified address. You may be encountering authentication issues with the SMTP server, blocked ports, or an invalid recipient email address – these are common reasons why an email may not be received after the GeoEvent Extension has successfully sent a message to an SMTP server. Have you tried using an external SMTP server rather than one available within your local network? You can configure the Send an Email output to use Google’s SMTP server (smtp.gmail.com) to send email notifications. You will need a Gmail account to do this as you will be required to provide authentication credentials which the GeoEvent Extension will send to the Google server via SSL. Note in the illustration above that the port number specified (465) is the one required when using Google’s SMTP server. Also note that while I can configure GeoEvent to reach out to the server smtp.gmail.com on port 465, our firewall here at Esri blocks outbound communication on that port, so I have to take my laptop off-campus if I want to use Google’s external SMTP Server. You will need to contact your IT administrator to identify the port(s) over which SMTP messages can be sent in your organization. Some organizations use port 25 and allow anonymous authentication; other common SMTP ports are 2525 and 587. You must also determine whether authentication credentials are required by your SMTP server (as they are for Google’s server). Have you looked in the karaf.log where GeoEvent writes its log messages to see if there are any logged messages from the SmtpOutboundTransport component? If GeoEvent were logging messages that it could not connect to the specified SMTP host, for example, that would give us something to investigate. You might try setting DEBUG logging for the com.esri.ges.transport.smtp.SmtpOutboundTransport component, to see if you can get GeoEvent to produce any additional logging messages. Finally, there are a couple of on-line tools / references which allow you to test SMTP connections. These were recommended to me, but I have not used them myself. Links to them are below if you would like to check them out. https://pingability.com/smtptest.jsp​ https://www.ndchost.com/wiki/mail/test-smtp-auth-telnet​ http://sourceforge.net/projects/smtp-test/ Hope this information helps – RJ ... View more

Hello Martin - I've been working with someone in Esri Technical Support who has had some success using the HTTP‑BasicAuthentication connector to poll a secure GeoRSS feed ... the difference being that he was using the RSS adapter whereas I assume you are using either the JSON or Generic-JSON adapter to poll your external server's service. Would you mind e-mailing me so that we can arrange either an on-line meeting or provide me the URL you are trying to poll and the credentials needed to access the external service? I'd like to try accessing the same resource you are trying to access in order to see if we can get this to work for you. Best Regards - RJ RJ Sunderman | Esri Professional Services Product Engineer | GeoEvent Extension for Server 380 New York St. | Redlands, CA 92373 | USA T 909-793-2853 x2065 | [email protected] ... View more

The new 10.3.x compilation of Twitter was uploaded to the product Gallery Friday 4/17. Here's a link to the Gallery item:  Twitter for GeoEvent The repository on GitHub has also been updated if you are looking for the source code. ... View more

A new compilation of the Twitter components for 10.3 was completed today which relies on a different library. I've run quick tests on this new offering and I think it addresses the issues described above. I will post an update to this thread once the new versions have been uploaded to the product Gallery. Best Regards - RJ ... View more

Hello Munhwan - Sorry that no one has gotten back in touch with you on your question. If you found a solution in the meantime and wanted to share it here, I'm sure the community would appreciate it. If your concern were that features were not being added to a feature service, I would suggest that you set the DEBUG level on the com.esri.ges.transport.featureService.FeatureServiceOutboundTransport. From within GeoEvent Manager, navigate to Logs > Settings and specify the name of the component for which you want to increase the logging level. A benefit of debug logging the Feature Service Outbound transport is that you'll see (in the karaf.log) all of the JSON transactions being made between GeoEvent and the ArcGIS Server REST interface used to add / update features. One problem I've seen is that a feature service with a field of type esriFiedlTypeString specifies a length restriction of say 5 characters. Several events are received by GeoEvent, one of whose Name field (a String) is longer than the 5 character restriction. GeoEvent will successfully adapt the events received, batch them up and send out a REST request to add / update features once every second. ArcGIS Server receives the request, observes that one of the event records has a String whose length it cannot accept ... and rejects the transaction. The result is that none of the events in that transaction come though to update features. You might find an indication of the rejection if you looked into the ArcGIS Server logs, but the only way you'd know from the GeoEvent karaf.log that there was a problem would be to look at DEBUG logging for the Feature Service Outbound transport and see the response received from the REST request GeoEvent made on the ArcGIS Server feature service's endpoint. Now - you indicated in your message that you observed events missing from both the feature service and the CSV system file you were using to log the processed events. This I cannot explain. Obviously an issue with ArcGIS Server rejecting one or more events because they do not meet the requirements of the feature service's schema would have nothing to do with writing that same event data out to a system file. Completely different transports and adapters are being used to support these two outputs. Were the event counts displayed in the GeoEvent Manager's Monitor page consistent? That is, did you receive 210 events In on an input, 420 events In/Out the GeoEvent Service and 210 events Out each your fs-out output and your text-csv-out output? About all I can recommend - apart from attaching a debugger and stepping through the execution of your custom transport and adapter - is to incorporate a system file JSON or CSV output directly off of your input and a second system file JSON or CSV output (writing to a different file) downstream, after any filtering / processing you might be doing ... just before you output the event data to your fs-out to add/update features. Then you can compare the content of the two output files. The data you see getting lost might be lost when an adapter fails to adapt a received payload of bytes, it might be discarded by a filter, it might be lost when a processor tries to process the data and throws an exception, or it might be lost when a REST transaction made by GeoEvent is rejected by the target server (as I describe above for adding / updating features). - RJ ... View more

Erik - We are making enhancements to the GeoEvent Simulator which should support the loading and simulation of data which uses <tab> characters to separate the different fields in an event. Please send an e-mail to the team alias [email protected] and request the Simulator update if you want to help test it out. - RJ ... View more

Generally speaking, each GeoEvent processed through a GeoEvent Service is independent. We don’t know when an event is going to arrive, and out-of-the-box processors do not cache event data – with a few exceptions. A Track Gap processor, for example, will cache the field tagged TRACK_ID and the date/time the last event with the cached TRACK_ID was observed … obviously so that it can produce an event when its timer expires and it determines that an event with TRACK_ID has not been seen in the specified time. A filter will cache some event information if it is configured with an ENTER or EXIT condition, since it has to know something about the previous event in order to determine if the event just received has entered / exited an area of interest. Our backlog of enhancements for 10.4 includes the concept of node-level caching, event aggregation, and temporal filtering. Big topics all, but enhancing the underlying implementation to provide node-level caching – so that any given processor / filter in a GeoEvent Service could interrogate a cache to retrieve a set of events previously received – would allow the development of an “aggregator” processor which could provide some basic statistical analysis: Send a single event out every 30 seconds whose Capacity field value is the largest observed Send a single event out every 30 seconds with an average of all of the observed Capacity values Sent a summary (or count) of all events received over the last 30 seconds which satisfy some attribute or spatial expression We’re eliciting requirements in these areas. I cannot commit that they will make the 10.4 release. My suspicion is that the 10.4 focus will be on increasing event throughput from “thousands” of events per second to “tens-of-thousands” of events per second. Obviously this presents a challenge to any sort of node-level caching enhancement. Just how much data to we expect GeoEvent to cache and which system resource (disk or memory) is going to take the hit? You might consider experimenting with the new spatial relationship operators available to filters – introduced at the 10.3.0 release. We tend to think of GeoFences as polygon areas then test to see if an event’s Geometry (a Point) is “inside” or “outside” the GeoFence. But the geometry of a GeoFence is not limited to a polygon. You could import a set of point features as your GeoFences and if the Geometry of a received event is a polygon, a filter could check to see which point GeoFences intersect the polygon event Geometry. A GeoTagger then, which is just a particular combination of a spatial filter and an event enricher, could produce a comma-separated list of the name of each point GeoFence which intersects a received polygon event Geometry. Here are the new spatial operators available for GeoEvent filtering (introduced with the 10.3 product release): Hope this information helps – RJ ... View more

Hello Erik - I suspect, from testing I’ve conducted, that the issue has something to do with the GeoEvent Simulator. I say this because I can drag/drop the same file I'm loading into the GeoEvent Simulator to a system folder being watched for CSV input and the Text adapter being used by both inbound connectors honors a specified Attribute Separator \u0009 (unicode value for <tab>) where as event data sent from the GeoEvent Simulator does not. For my test, I configured a tcp-text-in input: To create a GeoEvent Definition based on delimited data it receives Specifying an Attribute Separator as either \t or \u0009 … the event definition created by the input had only one event field, suggesting the attribute separator was not recognized However, if I configure a file-csv-in input with similar specifications .. the event definition created has the expected fields: Thank you for bringing this to our attention. I've submitted an issue to investigate the GeoEvent Simulator's handling of the tab delimited data. It's interesting that changing <tab> to semi-colon (;) in the input and specifying \u003B as the attribute separator works for both the GeoEvent Simulator and System File inputs. I suspect that the <tab> characters are being treated as some special form of whitespace by the Simulator. - RJ ... View more

Hello Jacob - To the best of my knowledge, no one on the GeoEvent product team has experience ingesting XML from Globalstar's SPOT Commercial Account Data Push service. I did perform a quick test to verify that the XML illustrated below, taken from the sample provided in the SPOT service's PDF, could be received and adapted by GeoEvent using a Receive XML on a REST Endpoint input configured as shown below. Looking briefly at the service's PDF, it appears that Globalstar supports HTTP, HTTPS, FTP, and SFTP. GeoEvent will not support FTP / SFTP. The extension generally supports RESTful data streams, not FTP based feeds. (There are inputs capable of watching a system folder for files, but these inputs have some significant limitations and are generally intended to prove that real-time analytics you've designed into a GeoEvent Service behave as you intend. Moving toward production we expect real-time data feeds to arrive via HTTP/POST or as replies to queries you make on an external server's URL.) Have you tried configuring the SPOT Data Push to provide you its XML data over HTTP? The out-of-the-box input Receive XML on a REST Endpoint introduced with the 10.3 product release should provide you that capability. That input creates a GeoEvent hosted REST endpoint which the external server should be able to reach via HTTP over port 6180 (e.g. http://myserver.domain:6180/geoevent/rest/receiver/name-of-input). Minimizing your server's exposure to attacks such as Denial of Service should be something we tackle after confirming that XML being posted to GeoEvent can be successfully received and adapted. We want to rule-out, for example, any issues with your organization's firewall which might prevent the SPOT service from reaching the GeoEvent REST endpoint. In the SPOT service's PDF, beneath the section titled Reducing the security vulnerabilities, I read that the receiver (GeoEvent in this case) can ignore the X-WSSE header information provided by the data feed. Once we confirm that GeoEvent can receive and adapt the XML being sent by the SPOT service, we can look at the possibility of using HTTPS to ensure data encryption and verifying data provided in the X-WSSE header. This may require some custom development on your part using the GeoEvent SDK to create a custom transport which implements the header validation illustrated in Appendix B – Sample Java Code to verify X-WSSE headers section of the SPOT servcie's PDF. Hope this information helps - RJ See also: ArcGeo Event Processor as a Callback URL​ Polling External Website for JSON via HTTPS ... View more