For the ‘Receive New Data Only’ property to work on the 'Poll an external website for ...' inputs, the external site has to implement that functionality via the response header 'Last-Modified' (see '14.29 Last-Modified'). If the external site doesn’t support/implement this response header then checking the ‘Receive New Data Only’ parameter 'Yes' won’t have any affect and all data that is returned by the external server will be processed.
Please also note, that setting this propert to 'Yes' does not cause the GeoEvent Input to set any request parameter/header to indicate to the external server that only new data should be returned. So it is up to the external website at the provided URL to implement the time window that new data is returned for (in addition to setting the 'Last-Modified' header value).
‘Receive New Data Only’ = No
Every response from the external server will be accepted and all data within each response will be processed.
‘Receive New Data Only’ = Yes & 'Last-Modified' IS Supported
IF 'Receive new Data Only' is 'Yes' AND the external server provides a response with a datetime value for the header name 'Last-Modified' (e.g. "Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT") THEN that lastModifiedDatetime will be used to determine if the response is processed or not:
- If lastModifiedDatetime is after the cachedLastModifiedDatetime the response data will be accepted and the input will set the cachedLastModifiedDatetime = lastModifiedDatetime.
- If lastModifiedDatetime is before the cachedLastModifiedDatetime the response data will be ignored.
Please note:
- it is up to the external server to respond with a subset of records that have been modifed since the lastModifiedDatetime. This might be done via
- The time window to filter modified results is entirely up to the external server.
- If the geoEvent polls every 30 seconds, but the server only returns modified results for the last 5 minutes, then the input may have to process 5 minutes worth of data every 30 seconds (assuming the data is changing more frequently than 30 seconds).
‘Receive New Data Only’ = Yes & 'Last-Modified' is NOT Supported
This situation will behave the same as setting the ‘Receive New Data Only’ = No. Unfortunately, the workarounds for this situation all depend on the external site:
- Modify the external server to support the 'Last-Modified' response header parameter.
- Implement on the server URL site a request URL parameter of ‘newvalues=true/false’
- If set to true, then the external server will return updated records only within an arbitrary time window.
- This is a bit messy because GeoEvent can’t control the polling/window of the last modified date.
- Another [potentially better] option would be to implement on the server URL site a request URL parameter for a defined window of updated values ‘newvalues=#’, where the numeric value references a defined time value (e.g. minutes).
- If omitted or <=0 all values are returned
- If >0 then any value that has been modified within that number of time units is returned
- This allows GeoEvent to control the polling/window of the last modified date.
- Use a filter in your GeoEvent Service to filter out older records
- This assumes the data returned by the server contains a date field that can be used as a "Last Modified" date (if not, you are out of luck)
- Add this filter immediately after your input to avoid processing unchanged records.
Finally
As a final note, there is no way to dynamically change the 'Parameters' property of the GeoEvent Input. So trying to set a dynamic/calculated datetime on the requet URL parameter will not work (e.g. 'Parameters' = "lastModifiedDate > ( now()+30)" ).
