Polling feature services for "Incremental Updates"

Blog Post created by rsunderman-esristaff Employee on Aug 20, 2015

Another recurring question:


I've configured a 'Poll an ArcGIS Server for Features' input to 'Get Incremental Updates', is there a way to prevent the input from polling all of the features in a feature class when GeoEvent Server is restarted?


The short answer is: No.  When GeoEvent Server is restarted (or the server on which it is running is rebooted), inputs which use the out-of-the-box FeatureService transport to poll an ArcGIS Server map service (or feature service) lose whatever value they've cached which enables them to query for features which are "new" relative to the last poll conducted by the input.


The capability to 'Get Incremental Updates' is unique to the 'Poll an ArcGIS Server for Features' input connector and should not be confused with the 'Receive New Data Only' parameter, exposed by the HTTP transport, which requires event data include the HTTP "Last-Modified" header. (Refer to comments in the thread Re: Receive RSS Inbound Connector.)


The issue we're exploring here deals only with the 'Poll an ArcGIS Server for Features' input connector -- or a custom input you may have configured which uses the FeatureService transport to poll an ArcGIS Server map / feature service.


An input configured to poll a map / feature service and retrieve only incremental feature updates maintains an in-memory cache. The value in this cache depends on whether your 'Method to Identify Incremental Updates' is ObjectID or Timestamp. In either case the input incorporates the largest value observed from its last poll into a WHERE clause so that only features whose OID (or date/time) is greater than the greatest value (from the last query) are returned (by the next query).


If you stop the input the cache is maintained, so that when the input is restarted it will be able to poll for features whose specified attribute value is greater than the value in the cache. If you stop the ArcGIS GeoEvent Server Windows service, or reboot the server, the cache is destroyed and the input has no way of knowing which features were polled previously. The next poll conducted by the input will retrieve all of the items in the map / feature service.


This becomes painfully obvious when one of the notification outputs (e.g. 'Send a Text Message' or 'Send an Email') are included in a GeoEvent Service which polls a map / feature service for event data. When the GIS Server is rebooted an e-mail recipient can potentially receive hundreds of messages if event data polled by the input satisfy filtering and/or processing criteria designed into a GeoEvent Service.


The motivation behind this behavior is that a cache persisted within a system file on disk could be difficult to find, might only be editable by a user with administrative credentials, and unnecessarily involves file I/O in a potentially high-volume event processing scenario. Locating and deleting a system file-based cache was deemed more burdensome than requiring that GeoEvent Server outputs be stopped in order to prevent unwanted notifications from being sent. Basically, this behavior is by design.


As a best practice, if you find you are frequently restarting GeoEvent Server (or having to reboot your server), make sure to stop all notification outputs, or any outputs you do not want to process events based on "old" features, when GeoEvent Server is restarted.


You can also employ a strategy of writing notification messages to a secondary feature layer, rather than directly to a notification output. The secondary feature layer acts as a notification message cache. You could then design a second GeoEvent Service (or extend your original GeoEvent Service) to poll this "notification message cache" and as event messages are sent to a notification output, use an 'Update a Feature' output to flag the notification message as having been sent. This will enable a filter to discard messages which have been sent and avoid sending repeat notifications.


If you have other approaches you have developed to deal with this particular behavior, your comments are welcome.


As always, I hope this information helps.


- RJ