Receive RSS Inbound Connector

Question

Hello GeoNet Community,

I am working through the RSS Tutorial for GeoEvent Processor and finding it difficult to receive new data only, is this possible if your RSS feed is missing the 'Last-Modified headers'?

For example, if we want to poll FEMA's Disaster RSS Feed for new data only (https://www.fema.gov/data/disasters.rss) could this be done in GeoEvent without introducing additional logic?

A few other approaches we may try are sending the RSS feed out to a feature service then back into GeoEvent, another option may be checking the 'PubDate' field in each entry for newly published data using a Field Calculator.

Katie

RJSunderman · Answer

Since the feed has no coordinate information, it doesn't make logical sense to add / update features in  a feature service. But we don't need to let that stop us.

I published a feature service with an attribute schema matching the feed's attributes. I imported an event definition from the feature service and deleted the objectid and geometry fields -- since I have no data for these coming from the feed.

I created an 'Update a Feature' output, using <guid> as the unique field for updates, and began using the feature service to cache data received from the RSS feed.

I noticed that the feed's data is advertising its date/time values as EST ... so they're placed into the feature class as UTC values (which have a a five hour offset from EST).

I was able to then configure a 'Poll an ArcGIS Server for Features' input to use the same tailored event definition (sans objectid and geometry) to obtain retrieve just the new records from the feature service.

Good work around - thanks for suggesting it.

- RJ

RJSunderman · Answer

Hello K -

Good to meet you out in San Diego at the UC. Here's some information I got from one of our developers on your question.

The 'Receive RSS' input is using standard HTTP header fields to determine whether or not data in a response to a poll is "new". Specifically, the input is relying on the "Last-Modified" header as described in Section 14 of RFC 2616, "Header Field Definitions", for the HTTP/1.1 protocol.

The input does not cache a date/time value like the 'Poll an ArcGIS Server for Features' input, so it cannot poll for data which is newer than a cached value. If FEMA's feed doesn't include the needed header the input will fail-over to poll for all data.

I suppose you could use a Field Calculator to obtain an event's receivedTime() and then compute a difference between that and the event's advertised publish date, using a filter to discard anything deemed "old" ... but that approach seems prone to error. You're probably going to receive duplicates you don't want and might filter events you actually do want.