topic Re: Poll an External Website for JSON using headers in ArcGIS GeoEvent Server Questions

Poll an External Website for JSON using headers

MarielaDel_Rio — Tue, 29 Jan 2019 00:10:18 GMT

How do I set up a GeoEvent input with headers that include a token?

Authorization: Bird ...TOKEN....

==================

No quotes on either side.

Re: Poll an External Website for JSON using headers

LeoLadefian5 — Wed, 06 Feb 2019 18:04:17 GMT

I'm doing the same thing, Did you try: APP-Version:3.0.0, Authorization:Bird {token}

Also, I think that API info pdf is confidential.

You shouldn't have to put the content type as it has it's own field in the settings?

Re: Poll an External Website for JSON using headers

MarielaDel_Rio — Wed, 27 Mar 2019 00:11:42 GMT

I got it working. I needed to include the Acceptable MIME types (Client Mode): application/vnd.mds.provider+json;

In the end, the URL was correct, but the header did not include double quotes or anything. Something like this:

Authorization:Bearer MY_TOKENsdafasdfasdfasdfsadf

Re: Poll an External Website for JSON using headers

SaraAbasi — Mon, 10 Feb 2020 13:57:40 GMT

Mariela,

I am setting up the same input. I appreciate it if you can share the screenshots of your Bird input connector as well as the Geoevent definition!

I did not find any esri document online for Bird. Thank you!

Re: Poll an External Website for JSON using headers

SaraAbasi — Mon, 10 Feb 2020 14:19:43 GMT

I got it working, I just need to make sure that the GeoEvent definition is correct.

Re: Poll an External Website for JSON using headers

SaraAbasi — Mon, 10 Feb 2020 15:00:50 GMT

Mariela,

Did you create a flat Geoevent definition? did you use any processor in the service?

This is the Gegeoevent definition created automatically. How can I get the coordinates to wok in the format of x and y!

Re: Poll an External Website for JSON using headers

MarielaDel_Rio — Mon, 10 Feb 2020 16:34:18 GMT

What are you trying to capture? It looks to me you are trying to capture the Status Change (MDS). We do not use status change because only shows what is available, not what is being used.

What we capture with geoEvent is the free_Bike_status from the GBFS API. The MDS (either trips or status_changes) cannot be captured because they come in pages and GeoEvent cannot paginate, the free_bike_status will give us what is available (same as status_change). We no longer use the status change method because it does not show "in use" scooters, and, like I mentioned, for available scooters, we have the Free_bike_status.

free_bike_status method (GBFS): the definition that was automatically created had Lat/Lon. After the definition was created, we added the geometry field to it

and marked Create GeoEvent Definition as "No".

Note that we used the JSON element Named "Bikes".

Our GeoEvent services looks like this:

The birdMapper has the Definition we created, and the feature service definition. This is where the geometry is calculated.

In the GeoEvent services, we indicate the geometry field so it can be calculated. This is separate from the X and Y, long/lat because we want a feature class created automatically, otherwise it would be a table with X Y that has to be converted.

I am not sure if that answers your questions, but for the MDS, we use python scripts that execute the URL and get the next page from the payload they retrieve. It is in a loop until it reaches the last page, at which time, we do the same for the next vendor and so on. Some vendors have expiring tokens (two vendors actually). For these, we read the token from a text file, we check for error codes in the script (look for token expired) and generate a new token when this happens (we call the token URL provided by the vendor), and save it into the file. The token can expire while running the script but we save it into a file to reuse it if it is still valid next time we run the script.

The trips script runs every night for the previous two days (from current day - 2 days to previous day). Vendors say they update everything within 12 hours, but we have found after two days, we still get older trips. We runt he trips scripts every night for the previous 48 to 12 hours and we also run it every morning for the previous 5 days. That way if there are changes or older trips that did not make it, will be included.

We also have one vendor who has an expiring token with the GBFS. This is the only vendor that has a token. The vendor reached out to ESRI and ESRI created a customized inbound Transport that has HTTP OAuth2 to generate a new token on the fly for that specific vendor.

Let me know if this helps. I can also share the python scripts if you need them.

Re: Poll an External Website for JSON using headers

SaraAbasi — Mon, 10 Feb 2020 19:18:11 GMT

Mariela, thank you for taking the time to answer my question, that definitely helped.

Do you use the python scripts on this page for the MDS? GitHub - openmobilityfoundation/mobility-data-specification: A data standard to enable communication between mobility co…

I am not sure how is it possible to send a message to your inbox to get the python scripts.

Re: Poll an External Website for JSON using headers

EricIronside — Wed, 22 Apr 2020 17:18:56 GMT

On March 1, MDS added a /vehicles end point to their specification for release 0.4.1. It should provide a cleaner way to get data without the paging start/end dates. Not sure if/when this will show up in vendor APIs.

https://github.com/openmobilityfoundation/mobility-data-specification/pull/376

Best,

Eric

Re: Poll an External Website for JSON using headers

MarielaDel_Rio — Wed, 22 Apr 2020 21:03:03 GMT

I know it has been a while since your post, but I will be more than happy to share the scripts.

What we do is, we have schedule them to run daily and weekly. The daily script will run at midnight and includes trips for the 36 hours prior to the date. What we found out is that, even though MDS is somewhat real time, trips keep coming after 12 hours (all vendors assure trips are updated no more than 5 minutes after they occur). We also noticed, when running the script as a test for previous weeks, we get trips we did not get the first time, hence, the weekly run. We run the previous 36 hours (there is overlapping), and also once a week, we run it for the previous 7 days (we do get new trips during this time).

Let me know if you want me to send you anything. They are python scripts that use the URL feeds, and tokens. It also has token renewal for those that need it.

Re: Poll an External Website for JSON using headers

MarielaDel_Rio — Wed, 22 Apr 2020 21:05:04 GMT

What I would like is for GeoEvent to handle variable number of points in the routes we pull, pagination, and token renewal out of the box. Right now, ESRI created a custom process to change the token once it expires. It was done in 10.6 but it worked on 10.7.1 and it works in 10.8 now

Re: Poll an External Website for JSON using headers

RJSunderman — Wed, 22 Apr 2020 23:35:06 GMT

Hello Mariela Del Rio –

When you say automatic handling of "variable number of points in the routes", I'm not sure I follow what you are asking for. Looking over the GeoEvent Definition illustrations in the thread I don't see an attribute for a route or routes that is multicardinal (indicating the attribute is an array containing more than one value).

Sara's reply to you on 10-February has a GeoEvent Definition whose event_time attribute looks kind of like a geoJSON geometry structure. The geoJSON specification for geometries places coordinates in an array. Our Generic-JSON inbound adapter includes a parameter asGeoJSON which can be used to tell the adapter to try parsing a field whose declared type is Geometry as a geoJSON geometry. The rest of the JSON structure is still handled as generic JSON, but any supposed geometry will be parsed differently, specifically so arrays of coordinate values can be interpreted as polylines or polygons. I'd have to experiment some with actual data from these feeds, but if this is what you're referring to we might be able to make something work for you without writing any code.

If, on the other hand, you are looking for GeoEvent Server's inbound adapter to iterate over a variable number of JSON objects or discrete data values in an array ... that isn't something we have included in our product road map. GeoEvent Server's support for hierarchy and multicardinality is actually pretty limited and we drew the line quite some time ago at providing some sort of configurable iterator to iterate over a variable list of values in order to perform processing on each value. There are some really powerful things you can do using regular expressions and a series of Field Calculator processors. You might check out this thread for an example.

Switching to your request for pagination for web service service(s), this is a pretty broad topic. There is no standard I'm aware of for how a given external web service might elect to communicate to a client that data retrieval should be performed as a series of queries rather than receiving all of the data as part of a single response to a single query. For example, the ArcGIS REST API for feature services specifies that a client should interpret a feature service returning exceededTransferLimit=true to mean that "there are more query results" and "you should continue to page through the results". GeoEvent Server is able to page through Esri's feature services when querying for a large number of feature records, but I'm not sure how we would implement a general solution for paging through any external web service's content. (I'm actually more familiar with the opposite, when a web service wants to return tens-of-megabytes of data to a client in response to a query and GeoEvent administrators ask how to configure GeoEvent Server to handle such a massive slug of data.)

Your third ask, for automatic token renewal, is part of our product backlog. What we want to do is retire, at least, the obsolete "OAuth1" authentication support offered by inbound transports and implement new support for OAuth2 authentication, which I think would include automatic token renewal. Our last research spike into this showed that while OAuth2 is a standard, there are subtle ways a service author can define how the different authorization types are implemented. We can probably develop a connector which supports the OKTA standard of OAuth2, but the effort was put on hold. When discussing this problem with customers I tell them that this is one of the cleanest examples for recommending a custom transport that I can think of. There are a bunch of different ways a web service might ask a client to first authenticate and acquire a token so that a second request (with the token) can be made authorizing a client's request. I don't recall any of my team helping to develop the custom inbound transport (or processor?) you refer to, but you are correct, if the development was completed using the 10.6 version of the GeoEvent Server Java SDK, you should be safe to continue using it for 10.7.x and 10.8.x ... we've no plans for breaking changes to the API which would require the custom component you had Esri build for you to need to be re-compiled.

Hope this information is helpful –
RJ

Re: Poll an External Website for JSON using headers

SaraAbasi — Fri, 29 May 2020 16:37:28 GMT

Thanks Mariela,

I appreciate you help and sharing the scripts.

I don't think it is possible to sent you a private message in Geonet community so I sent you a request in Linkedin.

Re: Poll an External Website for JSON using headers

STan_RWC — Wed, 12 Jan 2022 00:59:59 GMT

Hi Mariela,

I know that it has been awhile since your post, but I would like to know if you can still share the scripts.

Again, thank you for your posts on this topic. It was quite important to know that GeoEvent does not handle paginated data (in a loop/or dynamic matter) at this time. A lot of vendors that make their data available via open source design the output this way simply for security and limit overload from scraping. So appreciated that the get request loops still needs to be handled outside of GeoEvent.

Thank you,

Sherie