Latest Contributions by RJSunderman

Hello Andreas - My apologies that no one has gotten back with you in response to your question. GeoEvent was designed to rely on GIS-tier communications with an ArcGIS Server site using Server generated tokens. Support for web-tier authentication and authorization using SAML and IWA is being developed, but is limited, and is being introduced beginning with the 10.3.1 product release. I need to clarify your question. Are we working with two different GIS Server machines, both with ArcGIS Server installed, which are participating in different ArcGIS Server sites? I ask because you mentioned the term "cluster" and I want to make sure that the GIS Servers we are considering are not part of a single ArcGIS Server site cluster. If the two GIS Servers were participating in the same ArcGIS Server site cluster with the GeoEvent Extension installed on one GIS Server but not on the other we would be working with an unsupported configuration. All of the GIS Servers in a site's cluster must be provisioned exactly the same - this is important when using the GeoEvent Extension. Next question - is it an option for your first ArcGIS Server, the one which has the GeoEvent Extension installed, to reach the second ArcGIS Server without going through the Web Adapter? Design assumptions built-in to the GeoEvent 10.3 release (and previous releases) assume that GeoEvent will be able to discover services using the ArcGIS REST Services Directory and that the directory will be accessed through either the open REST endpoint http://server‑name.domain:6080/arcgis or the secure endpoint https://server‑name.domain:6443/arcgis. The preference would be that GeoEvent not attempt to go through the web adapter to discover a server's services. Some support for IWA was implemented in the 10.3.1 product release for registering ArcGIS Server connections as GeoEvent Data Stores. For example, you can specify that GeoEvent should use a token to access the data store and specify the secure endpoint (https://server‑name.domain:6443/arcgis) as the URI to be used. When the token expires, you will have to refresh your Data Store configuration with a new token in order to continue using the data store. You can also, beginning with the 10.3.1 release, select to use web-tier authentication and supply a recognized IWA username and password when registering a data store. My understanding is that, even when using web-tier authentication, that access must still be made through fully-qualified server-name / port URI ... either the http://server‑name.domain:6080/arcgis endpoint or the secure endpoint https://server‑name.domain:6443/arcgis. I will try to get someone to confirm this. Hope this information helps - RJ ... View more

Hello Felipe - Have you had a chance to look through the exercises in the Notifications in GeoEvent tutorial? The reason you see the event count increasing on your email-text-out output is that the output has successfully taken a GeoEvent from an internal queue, adapted the event data to produce a plain/text message, and sent that message to the specified SMTP Server as a request to generate an e-mail message to the specified recipient(s). This is a “fire-and-forget” type of operation. The GeoEvent Extension will not contact the specified SMTP server to determine its status, and there is no way for GeoEvent to process acknowledgement or error messages which might be sent from the SMTP server. You cannot assume that because an output incremented its event count that an email has been successfully generated, delivered, or received. The incremented event count only indicates that a message has been sent to the specified SMTP server for delivery to the specified address. You may be encountering authentication issues with the SMTP server, blocked ports, or an invalid recipient email address – these are common reasons why an email may not be received after the GeoEvent Extension has successfully sent a message to an SMTP server. Have you tried using an external SMTP server rather than one available within your local network? You can configure the Send an Email output to use Google’s SMTP server (smtp.gmail.com) to send email notifications. You will need a Gmail account to do this as you will be required to provide authentication credentials which the GeoEvent Extension will send to the Google server via SSL. Note in the illustration above that the port number specified (465) is the one required when using Google’s SMTP server. Also note that while I can configure GeoEvent to reach out to the server smtp.gmail.com on port 465, our firewall here at Esri blocks outbound communication on that port, so I have to take my laptop off-campus if I want to use Google’s external SMTP Server. You will need to contact your IT administrator to identify the port(s) over which SMTP messages can be sent in your organization. Some organizations use port 25 and allow anonymous authentication; other common SMTP ports are 2525 and 587. You must also determine whether authentication credentials are required by your SMTP server (as they are for Google’s server). Have you looked in the karaf.log where GeoEvent writes its log messages to see if there are any logged messages from the SmtpOutboundTransport component? If GeoEvent were logging messages that it could not connect to the specified SMTP host, for example, that would give us something to investigate. You might try setting DEBUG logging for the com.esri.ges.transport.smtp.SmtpOutboundTransport component, to see if you can get GeoEvent to produce any additional logging messages. Finally, there are a couple of on-line tools / references which allow you to test SMTP connections. These were recommended to me, but I have not used them myself. Links to them are below if you would like to check them out. https://pingability.com/smtptest.jsp​ https://www.ndchost.com/wiki/mail/test-smtp-auth-telnet​ http://sourceforge.net/projects/smtp-test/ Hope this information helps – RJ ... View more

Hello Martin - I've been working with someone in Esri Technical Support who has had some success using the HTTP‑BasicAuthentication connector to poll a secure GeoRSS feed ... the difference being that he was using the RSS adapter whereas I assume you are using either the JSON or Generic-JSON adapter to poll your external server's service. Would you mind e-mailing me so that we can arrange either an on-line meeting or provide me the URL you are trying to poll and the credentials needed to access the external service? I'd like to try accessing the same resource you are trying to access in order to see if we can get this to work for you. Best Regards - RJ RJ Sunderman | Esri Professional Services Product Engineer | GeoEvent Extension for Server 380 New York St. | Redlands, CA 92373 | USA T 909-793-2853 x2065 | rsunderman@esri.com ... View more

The new 10.3.x compilation of Twitter was uploaded to the product Gallery Friday 4/17. Here's a link to the Gallery item:  Twitter for GeoEvent The repository on GitHub has also been updated if you are looking for the source code. ... View more

A new compilation of the Twitter components for 10.3 was completed today which relies on a different library. I've run quick tests on this new offering and I think it addresses the issues described above. I will post an update to this thread once the new versions have been uploaded to the product Gallery. Best Regards - RJ ... View more

Hello Munhwan - Sorry that no one has gotten back in touch with you on your question. If you found a solution in the meantime and wanted to share it here, I'm sure the community would appreciate it. If your concern were that features were not being added to a feature service, I would suggest that you set the DEBUG level on the com.esri.ges.transport.featureService.FeatureServiceOutboundTransport. From within GeoEvent Manager, navigate to Logs > Settings and specify the name of the component for which you want to increase the logging level. A benefit of debug logging the Feature Service Outbound transport is that you'll see (in the karaf.log) all of the JSON transactions being made between GeoEvent and the ArcGIS Server REST interface used to add / update features. One problem I've seen is that a feature service with a field of type esriFiedlTypeString specifies a length restriction of say 5 characters. Several events are received by GeoEvent, one of whose Name field (a String) is longer than the 5 character restriction. GeoEvent will successfully adapt the events received, batch them up and send out a REST request to add / update features once every second. ArcGIS Server receives the request, observes that one of the event records has a String whose length it cannot accept ... and rejects the transaction. The result is that none of the events in that transaction come though to update features. You might find an indication of the rejection if you looked into the ArcGIS Server logs, but the only way you'd know from the GeoEvent karaf.log that there was a problem would be to look at DEBUG logging for the Feature Service Outbound transport and see the response received from the REST request GeoEvent made on the ArcGIS Server feature service's endpoint. Now - you indicated in your message that you observed events missing from both the feature service and the CSV system file you were using to log the processed events. This I cannot explain. Obviously an issue with ArcGIS Server rejecting one or more events because they do not meet the requirements of the feature service's schema would have nothing to do with writing that same event data out to a system file. Completely different transports and adapters are being used to support these two outputs. Were the event counts displayed in the GeoEvent Manager's Monitor page consistent? That is, did you receive 210 events In on an input, 420 events In/Out the GeoEvent Service and 210 events Out each your fs-out output and your text-csv-out output? About all I can recommend - apart from attaching a debugger and stepping through the execution of your custom transport and adapter - is to incorporate a system file JSON or CSV output directly off of your input and a second system file JSON or CSV output (writing to a different file) downstream, after any filtering / processing you might be doing ... just before you output the event data to your fs-out to add/update features. Then you can compare the content of the two output files. The data you see getting lost might be lost when an adapter fails to adapt a received payload of bytes, it might be discarded by a filter, it might be lost when a processor tries to process the data and throws an exception, or it might be lost when a REST transaction made by GeoEvent is rejected by the target server (as I describe above for adding / updating features). - RJ ... View more

Erik - We are making enhancements to the GeoEvent Simulator which should support the loading and simulation of data which uses <tab> characters to separate the different fields in an event. Please send an e-mail to the team alias geoevent@esri.com and request the Simulator update if you want to help test it out. - RJ ... View more

Generally speaking, each GeoEvent processed through a GeoEvent Service is independent. We don’t know when an event is going to arrive, and out-of-the-box processors do not cache event data – with a few exceptions. A Track Gap processor, for example, will cache the field tagged TRACK_ID and the date/time the last event with the cached TRACK_ID was observed … obviously so that it can produce an event when its timer expires and it determines that an event with TRACK_ID has not been seen in the specified time. A filter will cache some event information if it is configured with an ENTER or EXIT condition, since it has to know something about the previous event in order to determine if the event just received has entered / exited an area of interest. Our backlog of enhancements for 10.4 includes the concept of node-level caching, event aggregation, and temporal filtering. Big topics all, but enhancing the underlying implementation to provide node-level caching – so that any given processor / filter in a GeoEvent Service could interrogate a cache to retrieve a set of events previously received – would allow the development of an “aggregator” processor which could provide some basic statistical analysis: Send a single event out every 30 seconds whose Capacity field value is the largest observed Send a single event out every 30 seconds with an average of all of the observed Capacity values Sent a summary (or count) of all events received over the last 30 seconds which satisfy some attribute or spatial expression We’re eliciting requirements in these areas. I cannot commit that they will make the 10.4 release. My suspicion is that the 10.4 focus will be on increasing event throughput from “thousands” of events per second to “tens-of-thousands” of events per second. Obviously this presents a challenge to any sort of node-level caching enhancement. Just how much data to we expect GeoEvent to cache and which system resource (disk or memory) is going to take the hit? You might consider experimenting with the new spatial relationship operators available to filters – introduced at the 10.3.0 release. We tend to think of GeoFences as polygon areas then test to see if an event’s Geometry (a Point) is “inside” or “outside” the GeoFence. But the geometry of a GeoFence is not limited to a polygon. You could import a set of point features as your GeoFences and if the Geometry of a received event is a polygon, a filter could check to see which point GeoFences intersect the polygon event Geometry. A GeoTagger then, which is just a particular combination of a spatial filter and an event enricher, could produce a comma-separated list of the name of each point GeoFence which intersects a received polygon event Geometry. Here are the new spatial operators available for GeoEvent filtering (introduced with the 10.3 product release): Hope this information helps – RJ ... View more

Hello Erik - I suspect, from testing I’ve conducted, that the issue has something to do with the GeoEvent Simulator. I say this because I can drag/drop the same file I'm loading into the GeoEvent Simulator to a system folder being watched for CSV input and the Text adapter being used by both inbound connectors honors a specified Attribute Separator \u0009 (unicode value for <tab>) where as event data sent from the GeoEvent Simulator does not. For my test, I configured a tcp-text-in input: To create a GeoEvent Definition based on delimited data it receives Specifying an Attribute Separator as either \t or \u0009 … the event definition created by the input had only one event field, suggesting the attribute separator was not recognized However, if I configure a file-csv-in input with similar specifications .. the event definition created has the expected fields: Thank you for bringing this to our attention. I've submitted an issue to investigate the GeoEvent Simulator's handling of the tab delimited data. It's interesting that changing <tab> to semi-colon (;) in the input and specifying \u003B as the attribute separator works for both the GeoEvent Simulator and System File inputs. I suspect that the <tab> characters are being treated as some special form of whitespace by the Simulator. - RJ ... View more

Hello Jacob - To the best of my knowledge, no one on the GeoEvent product team has experience ingesting XML from Globalstar's SPOT Commercial Account Data Push service. I did perform a quick test to verify that the XML illustrated below, taken from the sample provided in the SPOT service's PDF, could be received and adapted by GeoEvent using a Receive XML on a REST Endpoint input configured as shown below. Looking briefly at the service's PDF, it appears that Globalstar supports HTTP, HTTPS, FTP, and SFTP. GeoEvent will not support FTP / SFTP. The extension generally supports RESTful data streams, not FTP based feeds. (There are inputs capable of watching a system folder for files, but these inputs have some significant limitations and are generally intended to prove that real-time analytics you've designed into a GeoEvent Service behave as you intend. Moving toward production we expect real-time data feeds to arrive via HTTP/POST or as replies to queries you make on an external server's URL.) Have you tried configuring the SPOT Data Push to provide you its XML data over HTTP? The out-of-the-box input Receive XML on a REST Endpoint introduced with the 10.3 product release should provide you that capability. That input creates a GeoEvent hosted REST endpoint which the external server should be able to reach via HTTP over port 6180 (e.g. http://myserver.domain:6180/geoevent/rest/receiver/name-of-input). Minimizing your server's exposure to attacks such as Denial of Service should be something we tackle after confirming that XML being posted to GeoEvent can be successfully received and adapted. We want to rule-out, for example, any issues with your organization's firewall which might prevent the SPOT service from reaching the GeoEvent REST endpoint. In the SPOT service's PDF, beneath the section titled Reducing the security vulnerabilities, I read that the receiver (GeoEvent in this case) can ignore the X-WSSE header information provided by the data feed. Once we confirm that GeoEvent can receive and adapt the XML being sent by the SPOT service, we can look at the possibility of using HTTPS to ensure data encryption and verifying data provided in the X-WSSE header. This may require some custom development on your part using the GeoEvent SDK to create a custom transport which implements the header validation illustrated in Appendix B – Sample Java Code to verify X-WSSE headers section of the SPOT servcie's PDF. Hope this information helps - RJ See also: ArcGeo Event Processor as a Callback URL​ Polling External Website for JSON via HTTPS ... View more

Sorry Jim - I don't have much experience working with EDN licenses, so I'm not able to advise you on how to determine what level of EDN licensing Avista has acquired. I'm going to pass this inquiry along with your contact information to Esri Customer Service. Someone should get back with you shortly. - RJ ... View more

Hello Jacob - You should be able to specify a URL parameter which includes embedded username and password credentials when configuring a Poll an External Website for XML inbound connector. However, you should be aware that doing poses a security risk, potentially exposing your credentials as the URL parameter is readable by anyone able to access your Server's REST endpoints ... specifically the GeoEvent endpoints on port 6180 (e.g. http://MyServer.MyDomain:6180/geoevent/rest). The better approach would be to configure a new inbound connector using the HTTP‑BasicAuthentication transport and XML adapter. The transport and adapter are both available out of the box. Configuring a new connector does not require any SDK development. From within GeoEvent Manager, navigate to Site > GeoEvent > Connectors and click Create Connector. In a separate tab/window you can click to open any of the existing connectors to see how they have been configured and use them as an example. If you do decide to use the out-of-the-box Poll an External Website for XML inbound connector, you may find that you need to URL encode the credentials you are embedding in the connector’s URL property. The correct syntax, I believe, is https://<user>:<password>@app.site.com/ So, for example, you would URL encode your string as follows: (Plain Text):  https://Jacob26:password123@app.site.com/feeds/assets.xml (URL Encoded):  https%3A%2F%2FJacob26%3Apassword123%40app.site.com%2Ffeeds%2Fassets.xml I tend to use http://meyerweb.com/eric/tools/dencoder/ when I need to interactively encode / decode URL strings. Looking at the examples you posted … you may also find that you need to drop the @domain.com suffix from the URL. I am less sure about this than recommending you URL encode the value however. It may be that if you URL encode the value then you’ll be able to specify the fully-qualified credentials:  user@domain.com:password@app.site.com Hope this information helps - RJ ... View more

Hello Felipe - The GeoEvent Extension does not provide an outbound connector specifically for the Whatsapp instant messaging application. It appears that Whatsapp is an XMPP based application and GeoEvent does include an XMPP adapter. You will probably find that you have to configure a custom connector using the out-of-the-box MessageFormatter adapter and XMPP transport however to get GeoEvent notifications to work with your Whatsapp application. In GeoEvent Manager, navigate to Site > GeoEvent > Connectors and look for the Send an Instant Message outbound connector (one of the connnectors available out-of-the-box). You can edit this connector to review its configuration and use that as a sample for configuring your own custom outbound connector. There is a section in the Notifications Tutorial​ which discusses how the Send an Instant Message connector is used. The exercises uses the Openfire / Spark XMPP client/server from igniterealtime.org. Installing these might help you figure out how Whatsapp needs to be configured to work with the custom XMPP outbound connector you configure for GeoEvent. Hope this information is helpful - RJ ... View more

James - My apologies that no one has gotten back with you in response to your questions. If you are able to install the GeoEvent Extension, but cannot load the GeoEvent Manger web application when browsing to https://localhost:6143/geoevent/manager and want to check product licensing, there is a run-checklicense.bat script in the ...\Program Files\ArcGIS\Server\GeoEvent\bin folder. Edit the script so that it appears as follows (new lines frequently do not register for me correctly in the installed file): echo off set output= set JAVA_EXE=C:\Program Files\ArcGIS\Server\framework\runtime\jre\bin\java FOR /F "delims=" %%i IN ('"%JAVA_EXE%" -cp .\PostInstaller.jar com.esri.geoevent.license.LicenseStatusCheck') DO set output=%%i if [%output%] == [] GOTO NOT_LICENSED echo License is valid exit /b 1 :NOT_LICENSED echo License not valid exit /b 0 The script should run the java.exe located in the JRE bundled with your ArcGIS Server installation, loading the specified *.jar file and executing the specified LicenseStatusCheck to verify your GeoEvent Extension has been correctly licensed. Hope this information helps - RJ ... View more