topic Re: Questions on Poll an external website for JSON in ArcGIS GeoEvent Server Questions

Questions on Poll an external website for JSON

wizgis — Thu, 23 Feb 2023 14:40:19 GMT

Hello All,

I have an external API that, on fetching, gives me 1156 records, and I am using Poll an external website for JSON input connector. I am able to configure the input and can see the count ;however, after a certain period (time interval set by me), it again ingests those 1156 records and the count doubles and it keeps going on.

I was just wondering if this is expected behavior or there is any setting that I am missing. Ideally, I was expecting the count to increase incrementally as records increase through API and not double after a certain interval of time.

Below are some screenshots for reference:

2312_2.PNG1156_t.PNG

Re: Questions on Poll an external website for JSON

JeffSilberberg — Mon, 27 Feb 2023 21:54:33 GMT

Did you specify a Unique Key / Track id on your side.

Usually pulling data that is not a duplicate of the last transaction set requires a start from key or date/time so that you only get the new records. Optionally, you could (depending on how owns & controls the source) archive / delete ach transaction once you have pulled it. Assuming a unique value like a sequence number.

Re: Questions on Poll an external website for JSON

wizgis — Tue, 28 Feb 2023 04:46:59 GMT

@JeffSilberberg

I did specify a unique id with Track Id tag however, was still running into issue.

On looking into this further I came across this link https://community.esri.com/t5/arcgis-geoevent-server-blog/receive-new-data-only-on-a-poll-an-external/ba-p/902042 and looks like the external link that we are using to Poll in records needs to have Last Modified Header and then only 'Receive New Data Only' will take in new records.

Re: Questions on Poll an external website for JSON

TomPaschke — Mon, 06 Mar 2023 16:58:12 GMT

@wizgis is there a way to specify a filter on the API side? e.g. time based or records of the last 5 min only?

I used different API implementations and depending on the available filter options at the source aka. the API I always tried to reduce the data volume that way as much as possible. The Last Modified Header is unfortunately very specific and not widely used.

Re: Questions on Poll an external website for JSON

wizgis — Tue, 07 Mar 2023 04:32:42 GMT

Thank you @TomPaschke for your inputs and letting me know that Last Modified Header property is not widely used. I'll have to check if there's a way to specify a filter on API side.