Poll an External Website for JSON using headers

759
12
01-28-2019 04:10 PM
Highlighted
New Contributor III

How do I set up a GeoEvent input with headers that include a token?

Authorization: Bird  ...TOKEN....

==================

No quotes on either side.

Reply
0 Kudos
12 Replies
Highlighted
Occasional Contributor II

I'm doing the same thing, Did you try:  APP-Version:3.0.0, Authorization:Bird {token}

Also, I think that API info pdf is confidential.

You shouldn't have to put the content type as it has it's own field in the settings?

Highlighted
New Contributor III

I got it working. I needed to include the Acceptable MIME types (Client Mode):  application/vnd.mds.provider+json;

In the end, the URL was correct, but the header did not include double quotes or anything. Something like this:

Authorization:Bearer MY_TOKENsdafasdfasdfasdfsadf

Highlighted
New Contributor III

Mariela,

I am setting up the same input. I appreciate it if you can share the screenshots of your Bird input connector as well as the Geoevent definition! 

I did not find any esri document online for Bird. Thank you!

Reply
0 Kudos
Highlighted
New Contributor III

I got it working, I just need to make sure that the GeoEvent definition is correct.

Highlighted
New Contributor III

Mariela,

Did you create a flat Geoevent definition? did you use any processor in the service?

This is the Gegeoevent definition created automatically. How can I get the coordinates to wok in the format of x and y!

Reply
0 Kudos
Highlighted
New Contributor III

What are you trying to capture? It looks to me you are trying to capture the Status Change (MDS). We do not use status change because only shows what is available, not what is being used.

What we capture with geoEvent is the free_Bike_status from the GBFS API. The MDS (either trips or status_changes) cannot be captured because they come in pages and GeoEvent cannot paginate, the free_bike_status will give us what is available (same as status_change). We no longer use the status change method because it does not show "in use" scooters, and, like I mentioned, for available scooters, we have the Free_bike_status. 

free_bike_status method (GBFS): the definition that was automatically created had Lat/Lon. After the definition was created, we added the geometry field to it 

and marked Create GeoEvent Definition as "No".

Note that we used the JSON element Named "Bikes". 

Our GeoEvent services looks like this:

The birdMapper has the Definition we created, and the feature service definition. This is where the geometry is calculated.

In the GeoEvent services, we indicate the geometry field so it can be calculated. This is separate from the X and Y, long/lat because we want a feature class created automatically, otherwise it would be a table with X Y that has to be converted.

I am not sure if that answers your questions, but for the MDS, we use python scripts that execute the URL and get the next page from the payload they retrieve. It is in a loop until it reaches the last page, at which time, we do the same for the next vendor and so on. Some vendors have expiring tokens (two vendors actually). For these, we read the token from a text file, we check for error codes in the script (look for token expired) and generate a new token when this happens (we call the token URL provided by the vendor), and save it into the file. The token can expire while running the script but we save it into a file to reuse it if it is still valid next time we run the script.

The trips script runs every night for the previous two days (from current day - 2 days to previous day). Vendors say they update everything within 12 hours, but we have found after two days, we still get older trips. We runt he trips scripts every night for the previous 48 to 12 hours and we also run it every morning for the previous 5 days. That way if there are changes or older trips that did not make it, will be included.

We also have one vendor who has an expiring token with the GBFS. This is the only vendor that has a token. The vendor reached out to ESRI and ESRI created a customized inbound Transport that has HTTP OAuth2 to generate a new token on the fly for that specific vendor.

Let me know if this helps. I can also share the python scripts if you need them.

Reply
0 Kudos
Highlighted
New Contributor III

Mariela, thank you for taking the time to answer my question, that definitely helped.

Do you use the python scripts on this page for the MDS? GitHub - openmobilityfoundation/mobility-data-specification: A data standard to enable communication... 

I am not sure how is it possible to send a message to your inbox to get the python scripts.

Reply
0 Kudos
Highlighted
New Contributor III

I know it has been a while since your post, but I will be more than happy to share the scripts.

What we do is, we have schedule them to run daily and weekly. The daily script will run at midnight and includes trips for the 36 hours prior to the date. What we found out is that, even though MDS is somewhat real time, trips keep coming after 12 hours (all vendors assure trips are updated no more than 5 minutes after they occur). We also noticed, when running the script as a test for previous weeks, we get trips we did not get the first time, hence, the weekly run. We run the previous 36 hours (there is overlapping), and also once a week, we run it for the previous 7 days (we do get new trips during this time).

Let me know if you want me to send you anything. They are python scripts that use the URL feeds, and tokens. It also has token renewal for those that need it.

Highlighted
New Contributor III

Thanks Mariela, 

I appreciate you help and sharing the scripts.

I don't think it is possible to sent you a private message in Geonet community so I sent you a request in Linkedin. 

Reply
0 Kudos