Latest Contributions by RJSunderman

‎05-21-2015

Sharon - The log messages you provide above look to me like the Rabbit MQ message broker service (provided by ArcGIS Server) has failed or is not running. Are you able to launch the GeoEvent Manager web application? Can you create a simple TCP/Text input -> TCP/Text output GeoEvent Service? If you send event data to the TCP input socket using GeoEvent Simulator, does the input's event count increment, but the 'In'/'Out' event count for the GeoEvent Service and event count on the output remain at zero (do not increment)? There are some trouble shooting steps I can share with you that are found in the product tutorial for clustering. Technically even if you are configuring a Server site with only a single server machine, you are still running a cluster - just with only one machine. Before we go into that, however, I'd like to get your feedback on the questions above. Thanks - RJ

‎05-21-2015

Hello Al / Alexander - My apologies that no one has replied to you on this post. The ArcGIS GeoEvent extension does not have an out-of-the-box connector for the Motorola ICC Pro. Generally GeoEvent does not provide connectors for specific commercial devices. When considering GeoEvent for real-time event processing the questions we need to ask are more general. How is the data being broadcast such that GeoEvent could receive it? Can data from the device be sent as an HTTP/POST to a REST endpoint hosted by GeoEvent, for example? Can the data be streamed over a TCP socket or WebSocket? Once we've figured out how the data will be delivered to GeoEvent, we need to know what format the data will be in. Will the data be broadcast in a generic XML structure, as generic JSON, or as comma-separated text, for example? There are adapters for the above data formats available out-of-the-box. If the ICC Pro were broadcasting its data in some other data format, then a custom adapter would need to be developed to interpret the data being sent by the device. A developer would use the provided GeoEvent SDK to develop the needed adapter. Al indicated that he was successful in getting the ICC Pro data written out to SQL Server tables. You might want to take a look at the thread GeoEvent process Oracle/SQL connector which describes some options for retrieving this data. Generally speaking, the Esri Feature Service is the interface GeoEvent relies on to access data in enterprise database tables. GeoEvent does not support direct database table queries out-of-the-box. We are preparing a response to a recent post How can I put a depedency in a custom processor. The response will likely be very technical and intended for developers. The developer in this case needs advice on incorporating a 3rd party JDBC component into the Apache Karaf OSGi container GeoEvent uses to manage the various components it uses during runtime. Hope this information is of some use to you - RJ

‎05-20-2015

This blog has been updated as part of a new series describing debugging techniques you can use when working to identify the root cause of an issue with a GeoEvent Server deployment or configuration. The original blog's text is included below, however, please consider the new blogs which can be accessed by clicking any of the quick links below: Debug Techniques - Configuring the application logger Debug Techniques - Add/Update Feature Outputs Debug Techniques - Application logging tips and tricks Debug Techniques - Geofence synchronization deep dive How to debug the Add a Feature and Update a Feature Output Connectors is probably the question I have been asked the most over the last couple of years working on GeoEvent Extension development team – so it’s appropriate that my inaugural blog to GeoNet address it. The scenario: An input appears to be successfully receiving and adapting the data from the stream and creating GeoEvents. The Filters and/or processors incorporated in a GeoEvent Service are handling the GeoEvents as expected. The event count on an Add a Feature or Update a Feature Output Connector is incrementing, but no features are being added or updated in the targeted feature layer. So, how do you start debugging to determine what the problem might be? My advice is to enable DEBUG logging on the feature service outbound transport to see if we can capture the JSON request being sent to ArcGIS Server and the response GeoEvent Extension receives from ArcGIS Server. I’ve attached an image below (FeatureServiceUpdate.png) of a karaf.log I created a while ago which shows the transactions taking place when a Output Connector performs an HTTP/POST to update features in a feature service. Don’t be concerned that the illustration identifies the 10.2.2 version – the concepts and workflows are the same for the GeoEvent 10.3 and 10.4 product releases when using a traditional RDBMS. To enable DEBUG logging on a single component in GeoEvent Manager: Navigate to the Logs page and click the Settings button. Enter the logging component in the text field Logger and select the DEBUG log level. Click Save. In this case you want to log DEBUG messages for the com.esri.ges.transport.featureService.FeatureServiceOutboundTransport component only. Setting a logging level of DEBUG on the ROOT component is not recommended. Doing this will produce a very verbose set of log messages and can cause the Logs page in the GeoEvent Manager to 'hang' as it tries to refresh the rapidly updating logs. With DEBUG logging enabled for the specified component, the GeoEvent Extension will produce more detailed logs when the feature service outbound transport handles event data. The DEBUG logging statements will include the JSON being sent and the ArcGIS Server’s response. I prefer looking at the log in a text editor, rather than using the log manager in GeoEvent Manager. You can find the karaf.log in the default folder C:\Program Files\ArcGIS\Server\GeoEventProcessor\data\log. In the FeatureServiceUpdate.png that is attached, find the two messages time stamped 2014-03-28 14:06:09,074. The “querying for missing track id XXXX” messages indicate the GeoEvent Extension has discovered that it has not cached any information on features with the TRACK_ID “SWA1568” or “SWA510”. Looking at the third message in the series, it shows the SQL where clause used to query the …/FeatureServer/0/query REST endpoint to discover the necessary OBJECTID values. The response from the ArcGIS Server includes a JSON array features[ ] with the geometry, OBJECTID, and unique identifier field (flightNumber in this example) for the features with the flight identifiers “SWA1568” and “SWA510”. Notice that it took 175 milliseconds for the GeoEvent Extension to receive the ArcGIS Server response: 14:06:09,250 - 14:06:09,75 = 75 ms You might find this query/response latency information valuable when profiling / debugging your GeoEvent Services which are adding or updating features through a feature service. Once the GeoEvent Extension has the necessary OBJECTID values for the features it wants to update, it posts a block of JSON to the …/FeatureServer/0/updateFeatures REST endpoint. ArcGIS Server responds with “success”:true 325 milliseconds later: (577 - 252 = 325). If you find too many JSON event records being included in the transactions, making the log file difficult to read, you can try configuring your Update a Feature Output Connector to specify that the ‘Maximum Features Per Transaction’ should be limited to 1 (the default is 500). This obviously not something you would do in a production environment, but while debugging it can make the log file much easier to read. If you find that the log is rolling over too frequently, you can edit settings in the following configuration file to allow the karaf.log to grow larger than 1MB and to keep more than 10 archival log files: ...\Program Files\ArcGIS\Server\GeoEvent\etc\org.ops4j.pax.logging.cfg (Click the thumbnail above to open a larger view) The logging package changed with the 10.6.0 release to use version 2.x of Log4J. Settings applicable for the log4j2.appender are illustrated in the attached ops4j.pax.logging.cfg.png file. You can also edit the message formatting specified by the layout.ConversionPattern in this configuration file to reformat the messages being written to the karaf.log - more information on that can be found here: http://www.codejava.net/coding/common-conversion-patterns-for-log4js-patternlayout Hope this information helps – RJ

‎05-20-2015

Hello Brian – Yes, one of the limitations of the File transport used by the ‘Watch a Folder for New CSV Files’ inbound connector is that it does not monitor file size or the date/time a file was last modified. The inbound connector will cache the name of any file it has previously read and will not re-read the contents of a file with that filename. This is by design; the connector does this to prevent ingesting event data which has already been processed. Edit: 23-Jan-2018 Behavior for the ‘Watch a Folder for New CSV Files’ inbound connector was changed at 10.5.1 to no longer require that a file’s name be changed for the input to consider it a new file. The mechanism watching the folder for new files still does not consider file properties such as changes to a file’s “last updated” timestamp or file size. However, if you want an input to re-read files you’ve placed in a folder, you can simply stop and restart your input connector and each file’s content will re-read with its content processed as newly received event records. The work your service is doing to download event content as a system file, uniquely name that file using a GUID, and place it in a system folder GeoEvent is watching should be sufficient to work around this limitation. We have an item open in our product backlog related to watching a system folder for JSON files in which files larger than about 5100 bytes are not ingested by the inbound connector watching the system folder. The root issue here may be similar to what you are observing - that some CSV files your service has copied to a system folder are not being picked-up by GeoEvent. There are several problems inherent when working with file-based input. One example is that GeoEvent may attempt to acquire a handle to a file before an external process is finished writing the file to disk. Another example is related to the issue above; GeoEvent needs to read the contents of a file into memory in “chunks” and may encounter a problem emulating a data stream when retrieving blocks of event data from a system file. Inputs which watch a system folder for files are generally intended to prove that event filtering and real-time analytics you have designed into a GeoEvent Service behave as you intend. They give you the ability to repeatedly send a small sample of event data to GeoEvent to test the behavior of your GeoEvent Service before transitioning to a production configuration in which real-time data feeds arrive via a stream (e.g. as an HTTP/POST from an external server, or as a reply to a query you make on an external server's URL). Are you able to identify specific CSV file content which GeoEvent is not retrieving? Is GeoEvent ever able to successfully retrieve data from those files? Or is it only occasionally skipping a file, and if you were to stop and restart the ‘Watch a Folder for New CSV Files’ input it would read the file the “second” time? If you can consistently reproduce the issue, please open an incident with Esri Technical Support. It would be helpful if you could provide samples of the data or help Tech Support reproduce the issue so that we can determine if you’re encountering a known limitation of the GeoEvent File transport or if there’s a bug we can address. Hope this information helps – RJ

‎05-07-2015

Hello Daniel - I see that from your post that you're working with Esri on mapping the fields from the feed you are receiving from Network Fleet in order to update features in a feature class. Esri Technical Support should be able to help you with this, but as a reference, you might want to take a look at Module 4 of the product introduction tutorial. Specifically pages 30-31 and 36-38 which illustrate how a Field Mapper is used to change the fields / schema of a GeoEvent. Mapping a GeoEvent from one schema or event definition to another so that you can update features through a feature service is primarily what the Field Mapper processor is used for. You can download the product tutorial from our Product Gallery. Here's a link to the tutorials: http://links.esri.com/geoevent-tutorials Hope this information helps - RJ

‎05-07-2015

Hello Thomas - What Daniel describes is certainly viable, and probably the preferred way to allow Verizon to HTTP / POST data to your ArcGIS Server on-premises. To answer your second question, GeoEvent is an extension to ArcGIS Server and requires ArcGIS Server be installed. So, no, you cannot install GeoEvent on a server which does not have ArcGIS Server. An alternative to having your IT staff create tunnels for Verizon IPs would be to stand-up an ArcGIS Server and GeoEvent Extension in the cloud, say on Amazon EC2. This server would be public facing. Because Web Sockets use port 80 you could then have your primary on-premises server connect to a Web Socket hosed by the public facing server in the cloud ... and then relay event information from the GeoEvent running in the cloud to the GeoEvent you have running behind your firewall. The downside to this approach is that you have to license ArcGIS Server and the GeoEvent Extension for both the server you have stood-up in the cloud and the server you have on-premises. That's why I think Daniel's approach would be preferred - but if opening a tunnel for Verizon to use to access your on-premises server is not an option, then at least you have an alternative. Hope this information helps - RJ

‎05-01-2015

Hello Larry - The incident and TrackGap GeoEvent Definitions are owned by their respective core processors - the Incident Detector and Track Gap Detector. While you were able to edit these event definitions prior to 10.3, it is not recommended. Each processor's implementation expects the GeoEvent Definition structure you see immediately following product installation. You can copy the out-of-the-box event definition and make any changes you need to your copy of the GeoEvent Definition: Just be sure to direct the GeoEvent output from your Incident Detector or Track Gap Detector through a Field Mapper processor to map the event data to a structure you want to use. Hope this information helps - RJ

‎05-01-2015

So why is it that the geoevent Extension stops after 4-6 hours? There was an issue we observed, primarily with updating feature services hosted via ArcGIS Online in the 10.3 release. A few users also observed the issue using local feature services they had published to their on-premises Server. The symptom was that GeoEvent would appear to be receiving and processing data (its event counters would increment), but updates to features through a feature service would stop after a variable length of time. We usually found messages in the karaf.log with the text Error: null response or Error Posting to URL ... java.io.IOException: null respons when this specific issue would manifest. The issue was addressed with the 10.3 Patch 1 for GeoEvent, available here from Esri Support. I am not sure if the issue was actually present in earlier releases (10.2.x). In any case, I'd encourage you to upgrade to the 10.3.1 release which should be publicly available in just a couple of weeks (13-May 2015). The focus for 10.3.1 was on improving the product's stability, however, we did complete a product enhancement with regard to registering an ArcGIS Server connection as a GeoEvent Data Store. You can now specify whether you want to use a token for authentication or if you want to use web-tier authentication and provide a username / password when registering a GeoEvent Data Store. This is new at the 10.3.1 release. If we connect to server 2 with the directy URL then we the error below: properly due a certificate error. Invalid URL: Please provide a valid url for example: http://<hostname>:<port>/arcgis There were enhancements in the area of security certificates introduced in both the 10.3 and 10.3.1 releases. GeoEvent should trust the ArcGIS Server certificate presented by the local server on which it was installed at the 10.3 release (rather than relying on a GeoEvent self-signed certificate). With the 10.3.1 release GeoEvent will also trust any SSL certificates you have imported using the ArcGIS Server Administrator Directory. After you've upgraded to the latest release, if you find that, after working for a period of time, feature updates stop working, you see log messages you believe are security or SSL certificate related, or need advice on configuring your enterprise's reverse proxy or web-tier authentication ... please go ahead and open new GeoNet discussion threads for the separate issues. Questions on enterprise system architecture and security configuration I will probably have to refer to Esri Tech Support. I'm better positioned to offer advice on questions concerning GeoEvent product functionality. Best Regards - RJ

‎04-30-2015

Hello Greg - I do not believe GeoEvent will support what you are trying to do for a couple of reasons. First, GeoEvent does not provide any sort of node-level cache which you can access.The "nodes" in a GeoEvent Service (inputs, outputs, filters, processors) generally do not cache information from events they receive. Every node handles the GeoEvents it receives independently. There are a few exceptions, but the information being cached in these cases is very specific to the node's function and is not accessible for you to extract or use. Second, I don't believe there is any syntax for variable substitution built-in to an inbound connector's configuration. If you take a look at my reply to Thread_510827 (which illustrates how to include an event field name in a spatial filtering expression), you'll see that there are a few places within GeoEvent that you can specify ${fieldName} in place of a literal string and GeoEvent will dynamically replace the "macro" ${fieldName} with a value taken from the GeoEvent currently being handled by that node. Because there is no variable substitution syntax built-in to an inbound connector's configuration, whatever text you enter to configure an input must be the literal text value. I'm trying to think if recommending you explore using the GeoEvent SDK to develop a custom transport would support what you are trying to do. I suppose it would be possible to implement a self-modifying transport which would specify, say, the URL used initially to poll an external server. Then, after data is received from a poll, if the transport can retrieve data values from the adapter, the transport could use one or more of the received data values values to - possibly conditionally - reconfigure itself before the next poll is conducted. Keep in mind, though, that it is the responsibility of an adapter to take the byte stream from the connector's transport and "adapt" it to create a GeoEvent. You are looking at using the SDK here to develop a custom adapter and transport, functionality found right up front before a GeoEvent is even passed to a GeoEvent Service for filtering / processing ... and I don't know if member variables can be accessed between a transport object's classes an adapter object's classes. The SDK may not even support what you are trying to do. Hope this information helps - RJ

‎04-30-2015

Hello Undral – I believe the functionality you are looking for is an enhancement being introduced in the 10.3.1 product. Let’s say you design a GeoEvent Service with a filter as illustrated below: In this example the GeoFence category is still specified as a constant String, AdminAreas. But the GeoFence name is dynamic: ${TrackID}. Notice the required decoration around the value indicating to GeoEvent that TrackID is the name of a field, not a literal string. In this example, only GeoEvents whose TrackID attribute value matches the name of the GeoFence the event’s Geometry is found to be INSIDE will be allowed through the filter. Rather than a tcp-text-out you would use an output suitable for notification such as Send an Email. Hope this information helps – RJ

‎04-29-2015

Felix, A colleague mentioned that while the Java AIS library https://github.com/tbsalling/aismessages is open source, it does require a commercial license. Since TB Salling is what James referenced in his post, I don't think James will be able to share the custom adapter he built - it's probably licensed for use by their client / customer. There are a few other Java AIS libraries out there: https://github.com/dma-ais/AisLib (Java 😎 https://github.com/schwehr/libais (built on a core C++ lib) Some other tools / hints you might find helpful: Use telnet to check the AIS stream. This downloads page from nmearouter.com provides a desktop application which can connect to an AIS stream and decode messages. The site http://catb.org/gpsd/AIVDM.html has good descriptions of all AIS messages. Some Python and Java AIS parsers built on a fast C++ core are available from https://github.com/schwehr/libais Some additional parsers can be found here: https://github.com/bcl/aisparser It was also mentioned that serial to TCP/IP hardware adapters were a big help to the development team. They allowed access the AIS transponders remotely. A big thank you to the Esri Australia team for the information above. You can reach out to them via their distributor page: https://esriaustralia.com.au/contact-us

‎04-28-2015

Hello Felix - The AIS data stream you illustrate looks like raw 7-bit encoded NMEA-0183HS AIS data. The GeoEvent Extension does not provide an adapter which is capable of decoding this format. NMEA-0183HS messages are a variation of the NMEA 0183 data standard. They are 7-bit encoded ASCII messages which are not human readable. While they are part of the broader NMEA specification, they are not what common consumer devices like Garmin or smart phone applications broadcast. The NMEA messages the GeoEvent Extension is able to ingest are 8-bit NMEA-0183 messages which typically look more like the following: The full NMEA specification details a variety of different message sentence structures. There was a pull-request, for example, on the Esri / nmea-for-geoevent GitHub project to modify the sample connector we have on the gallery to support RMC sentences of different lengths. Having open-sourced our NMEA implementation, developers are able to tailor the connector to match their device's sentence structure - if what they are receiving does not happen to match the GGA / GSA / GSV / RMC sentence structures illustrated above. Since the NMEA-0183HS AIS data you are receiving cannot be adapted by the NMEA connector on the Gallery, have you considered the exactEarth Latest Vessel Information connector available from the Partner Gallery? Best Regards - RJ

‎04-28-2015

The GeoEvent extension includes an inbound connector which is able to poll an existing Esri feature service, for example, to retrieve data from a feature layer. The feature service in this case acts as a data broker which the GeoEvent extension can work through to access the underlying feature data in the database. The product team has no plans to provide ODBC / JDBC connectors to retrieve data directly from RDBMS tables. Connectors such as these would be database specific and we are working to keep GeoEvent database agnostic. You do have a few options available, however, without relying on an ODBC / JDBC connector: Since ArcMap will not allow you to publish a map document as a feature service if the MXD has no feature layers, you cannot create an MXD with *only* a non-spatial table and publish a feature service. ArcMap will require that you include at least one feature layer whose "features" come from a "feature class" … a spatial table containing geometry. However, if you add your non-spatial table to a map document and then add any feature layer, even one pointing to an empty (dummy) feature class which has no features, you should be able to publish a feature service. An out-of-the-box GeoEvent input can then poll the non-spatial table for "features" through the feature service. When configuring the Poll an ArcGIS Server for Featuresinbound connector, you simply select the non-spatial table as the "layer" rather than the actual (dummy) feature layer which will be forever empty (no features). Another option is to create a script which would periodically retrieve the non-spatial event data from the database's table, and then HTTP/POST the data directly to an endpoint associated with a GeoEvent Extension input. The out-of-the-box inbound connectors expect either Generic JSON or Esri Feature JSON. Later releases of GeoEvent also support XML and geoJSON. You might also look into creating a wrapper around the RDBMS using an open API such as the OData protocol (http://www.odata.org/). The OData protocol provides rapid solution development incorporating database access through web services as a front facet to the backend database. We are only interested in retrieving row data as "events" … and the OData protocol provides an REST endpoint through which the database records can be retrieved as generic JSON. This approach fits well with the GeoEvent extension which provides a generic JSON adapter out-of-the-box. You can then configure a Poll an external website for JSON inbound connector to poll the OData endpoint for data – or configure a Receive JSON on a REST endpoint inbound connector to receive JSON data from an OData HTTP POST (though I'm not sure if OData is capable of retrieving the database records and POSTing them to an external client). An architecture for the OData approach might look something like the following: Hope this information is helpful - RJ

‎04-28-2015

ArcGIS Server administrators who have integrated either Portal or the Web Adaptor into their site and want to register a connection to their ArcGIS Server as a GeoEvent Data Store – in order to poll a feature service for features, for example – will want to know about the Esri Resource Proxy available on GitHub. The proxy configuration provides administrators relying on username and password authentication a work around for GeoEvent setups which need to remain at the 10.2.x / 10.3 product releases. Once you have upgraded to the 10.3.1 product release (publically available 13-May 2015) you should no longer need to deploy the proxy as GeoEvent will accept and retain your users’ credentials for you. The GitHub project provides proxy files for accessing resources secured with token based authentication (particularly important when working with the GeoEvent Extension) as well as accessing cross domain resources and accessing resources using OAuth 2.0 application login. The project provides proxies for the .NET, Java, and PHP web application frameworks. Once the resource proxy has been deployed within your firewall, you should harden access to the proxy so that only GeoEvent can access it. Additional information can be found on the https://github.com/Esri/resource-proxy project page.

‎04-28-2015

Andreas - Some feedback specifically on your question: "Why is the GEP trying to access an admin URL?" https://<Web adaptor URL>/arcgistest/admin/data/findItems?f=json&token=XXXXXXX&types=egdb: GET Request failed(HTTP/1.1 403 Forbidden). The GeoEvent Extension is using the admin API to determine if a managed geodatabase has been registered with ArcGIS Server. If there is a managed geodatabase, when configuring a new 'Send Features to a Stream Service' output and selecting to publish a Stream Service, you will have the option to check the 'Store Latest' checkbox and publish a "buddy" feature service which will cache the most recent observation for each received TRACK_ID. If you do not have a managed geodatabase registered with ArcGIS Server, the 'Store Latest' option is not available. - RJ

Online Status	Offline
Date Last Visited	Wednesday

My Ideas

Latest Contributions by RJSunderman

Re: Error log entry:

Re: GeoEvent server as OPC client?

Debugging the Add a Feature / Update a Feature Output Connectors

Re: Watching for new CSV Files

Re: GeoEvent Update Feature

Re: Geoevent Processor and Verizon Networkfleet

Re: Incident Detector Processor 10.3 GeoEvent Definition

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

Re: Poll External Website for JSON with variables

Re: One to One relationship between GeoFence and GeoEvent

Re: Ais decoder

Re: Ais decoder

Re: GeoEvent process Oracle/SQL connector

Esri resource proxy available on GitHub

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

Re: GeoEvent input connector using token authentic...

Re: Updating the Status of a Fire Unit

Re: Is there a way to delete records from a Featur...

Re: GeoEvent-Poll an External Website for JSON

Re: GeoFence sync is not detecting deleted feature...

ArcGIS Analyse