Poll an External Website for JSON using headers

3064
13
01-28-2019 04:10 PM
MarielaDel_Rio
New Contributor III

How do I set up a GeoEvent input with headers that include a token?

Authorization: Bird  ...TOKEN....

==================

No quotes on either side.

0 Kudos
13 Replies
STan_RWC
New Contributor II

Hi Mariela,

I know that it has been awhile since your post, but I would like to know if you can still share the scripts. 

Again, thank you for your posts on this topic. It was quite important to know that GeoEvent does not handle paginated data (in a loop/or dynamic matter) at this time. A lot of vendors that make their data available via open source design the output this way simply for security and limit overload from scraping. So appreciated that the get request loops still needs to be handled outside of GeoEvent. 

Thank you,

Sherie

0 Kudos
EricIronside
Esri Regular Contributor

On March 1, MDS added a /vehicles end point to their specification for release 0.4.1. It should provide a cleaner way to get data without the paging start/end dates. Not sure if/when this will show up in vendor APIs.

https://github.com/openmobilityfoundation/mobility-data-specification/pull/376  

Best,

Eric

0 Kudos
MarielaDel_Rio
New Contributor III

What I would like is for GeoEvent to handle variable number of points in the routes we pull, pagination, and token renewal out of the box. Right now, ESRI created a custom process to change the token once it expires. It was done in 10.6 but it worked on 10.7.1 and it works in 10.8 now

0 Kudos
RJSunderman
Esri Regular Contributor

Hello Mariela Del Rio

When you say automatic handling of "variable number of points in the routes", I'm not sure I follow what you are asking for. Looking over the GeoEvent Definition illustrations in the thread I don't see an attribute for a route or routes that is multicardinal (indicating the attribute is an array containing more than one value).

Sara's reply to you on 10-February has a GeoEvent Definition whose event_time attribute looks kind of like a geoJSON geometry structure. The geoJSON specification for geometries places coordinates in an array. Our Generic-JSON inbound adapter includes a parameter asGeoJSON which can be used to tell the adapter to try parsing a field whose declared type is Geometry as a geoJSON geometry. The rest of the JSON structure is still handled as generic JSON, but any supposed geometry will be parsed differently, specifically so arrays of coordinate values can be interpreted as polylines or polygons. I'd have to experiment some with actual data from these feeds, but if this is what you're referring to we might be able to make something work for you without writing any code.

If, on the other hand, you are looking for GeoEvent Server's inbound adapter to iterate over a variable number of JSON objects or discrete data values in an array ... that isn't something we have included in our product road map. GeoEvent Server's support for hierarchy and multicardinality is actually pretty limited and we drew the line quite some time ago at providing some sort of configurable iterator to iterate over a variable list of values in order to perform processing on each value. There are some really powerful things you can do using regular expressions and a series of Field Calculator processors. You might check out this thread for an example.  

Switching to your request for pagination for web service service(s), this is a pretty broad topic. There is no standard I'm aware of for how a given external web service might elect to communicate to a client that data retrieval should be performed as a series of queries rather than receiving all of the data as part of a single response to a single query. For example, the ArcGIS REST API for feature services specifies that a client should interpret a feature service returning exceededTransferLimit=true to mean that "there are more query results" and "you should continue to page through the results". GeoEvent Server is able to page through Esri's feature services when querying for a large number of feature records, but I'm not sure how we would implement a general solution for paging through any external web service's content. (I'm actually more familiar with the opposite, when a web service wants to return tens-of-megabytes of data to a client in response to a query and GeoEvent administrators ask how to configure GeoEvent Server to handle such a massive slug of data.) 

Your third ask, for automatic token renewal, is part of our product backlog. What we want to do is retire, at least, the obsolete "OAuth1" authentication support offered by inbound transports and implement new support for OAuth2 authentication, which I think would include automatic token renewal. Our last research spike into this showed that while OAuth2 is a standard, there are subtle ways a service author can define how the different authorization types are implemented. We can probably develop a connector which supports the OKTA standard of OAuth2, but the effort was put on hold. When discussing this problem with customers I tell them that this is one of the cleanest examples for recommending a custom transport that I can think of. There are a bunch of different ways a web service might ask a client to first authenticate and acquire a token so that a second request (with the token) can be made authorizing a client's request. I don't recall any of my team helping to develop the custom inbound transport (or processor?) you refer to, but you are correct, if the development was completed using the 10.6 version of the GeoEvent Server Java SDK, you should be safe to continue using it for 10.7.x and 10.8.x ... we've no plans for breaking changes to the API which would require the custom component you had Esri build for you to need to be re-compiled.

Hope this information is helpful –
RJ