Questions on Poll an external website for JSON

[wizgis]

Hello All, 

I have an external API that, on fetching, gives me 1156 records, and I am using Poll an external website for JSON input connector. I am able to configure the input and can see the count ;however, after a certain period (time interval set by me), it again ingests those 1156 records and the count doubles and it keeps going on.

I was just wondering if this is expected behavior or there is any setting that I am missing. Ideally, I was expecting the count to increase incrementally as records increase through API and not double after a certain interval of time. 

Below are some screenshots for reference: 

[JeffSilberberg]

Did you specify a Unique Key / Track id on your side. 

Usually pulling data that is not a duplicate of the last transaction set requires a start from key or date/time so that you only get the new records.  Optionally, you could (depending on how owns & controls the source) archive / delete ach transaction once you have pulled it.  Assuming a unique value like a sequence number. 

[wizgis]

@JeffSilberberg 

I did specify a unique id with Track Id tag however, was still running into issue. 

On looking into this further I came across this link https://community.esri.com/t5/arcgis-geoevent-server-blog/receive-new-data-only-on-a-poll-an-externa... and looks like the external link that we are using to Poll in records needs to have Last Modified Header and then only 'Receive New Data Only' will take in new records. 

[TomPaschke · Accepted solution]

@wizgis is there a way to specify a filter on the API side? e.g. time based or records of the last 5 min only?

I used different API implementations and depending on the available filter options at the source aka. the API I always tried to reduce the data volume that way as much as possible. The Last Modified Header is unfortunately very specific and not widely used. 

[wizgis]

Thank you @TomPaschke  for your inputs and letting me know that Last Modified Header property is not widely used. I'll have to check if there's a way to specify a filter on API side.