Select to view content in your preferred language

Poll an External Website for JSON using headers

3682
13
01-28-2019 04:10 PM
MarielaDel_Rio
Regular Contributor

How do I set up a GeoEvent input with headers that include a token?

Authorization: Bird  ...TOKEN....

==================

No quotes on either side.

0 Kudos
13 Replies
LeoLadefian5
Frequent Contributor

I'm doing the same thing, Did you try:  APP-Version:3.0.0, Authorization:Bird {token}

Also, I think that API info pdf is confidential.

You shouldn't have to put the content type as it has it's own field in the settings?

MarielaDel_Rio
Regular Contributor

I got it working. I needed to include the Acceptable MIME types (Client Mode):  application/vnd.mds.provider+json;

In the end, the URL was correct, but the header did not include double quotes or anything. Something like this:

Authorization:Bearer MY_TOKENsdafasdfasdfasdfsadf

SaraAbasi
Occasional Contributor

Mariela,

I am setting up the same input. I appreciate it if you can share the screenshots of your Bird input connector as well as the Geoevent definition! 

I did not find any esri document online for Bird. Thank you!

0 Kudos
SaraAbasi
Occasional Contributor

I got it working, I just need to make sure that the GeoEvent definition is correct.

SaraAbasi
Occasional Contributor

Mariela,

Did you create a flat Geoevent definition? did you use any processor in the service?

This is the Gegeoevent definition created automatically. How can I get the coordinates to wok in the format of x and y!

0 Kudos
MarielaDel_Rio
Regular Contributor

What are you trying to capture? It looks to me you are trying to capture the Status Change (MDS). We do not use status change because only shows what is available, not what is being used.

What we capture with geoEvent is the free_Bike_status from the GBFS API. The MDS (either trips or status_changes) cannot be captured because they come in pages and GeoEvent cannot paginate, the free_bike_status will give us what is available (same as status_change). We no longer use the status change method because it does not show "in use" scooters, and, like I mentioned, for available scooters, we have the Free_bike_status. 

free_bike_status method (GBFS): the definition that was automatically created had Lat/Lon. After the definition was created, we added the geometry field to it 

and marked Create GeoEvent Definition as "No".

Note that we used the JSON element Named "Bikes". 

Our GeoEvent services looks like this:

The birdMapper has the Definition we created, and the feature service definition. This is where the geometry is calculated.

In the GeoEvent services, we indicate the geometry field so it can be calculated. This is separate from the X and Y, long/lat because we want a feature class created automatically, otherwise it would be a table with X Y that has to be converted.

I am not sure if that answers your questions, but for the MDS, we use python scripts that execute the URL and get the next page from the payload they retrieve. It is in a loop until it reaches the last page, at which time, we do the same for the next vendor and so on. Some vendors have expiring tokens (two vendors actually). For these, we read the token from a text file, we check for error codes in the script (look for token expired) and generate a new token when this happens (we call the token URL provided by the vendor), and save it into the file. The token can expire while running the script but we save it into a file to reuse it if it is still valid next time we run the script.

The trips script runs every night for the previous two days (from current day - 2 days to previous day). Vendors say they update everything within 12 hours, but we have found after two days, we still get older trips. We runt he trips scripts every night for the previous 48 to 12 hours and we also run it every morning for the previous 5 days. That way if there are changes or older trips that did not make it, will be included.

We also have one vendor who has an expiring token with the GBFS. This is the only vendor that has a token. The vendor reached out to ESRI and ESRI created a customized inbound Transport that has HTTP OAuth2 to generate a new token on the fly for that specific vendor.

Let me know if this helps. I can also share the python scripts if you need them.

0 Kudos
SaraAbasi
Occasional Contributor

Mariela, thank you for taking the time to answer my question, that definitely helped.

Do you use the python scripts on this page for the MDS? GitHub - openmobilityfoundation/mobility-data-specification: A data standard to enable communication... 

I am not sure how is it possible to send a message to your inbox to get the python scripts.

0 Kudos
MarielaDel_Rio
Regular Contributor

I know it has been a while since your post, but I will be more than happy to share the scripts.

What we do is, we have schedule them to run daily and weekly. The daily script will run at midnight and includes trips for the 36 hours prior to the date. What we found out is that, even though MDS is somewhat real time, trips keep coming after 12 hours (all vendors assure trips are updated no more than 5 minutes after they occur). We also noticed, when running the script as a test for previous weeks, we get trips we did not get the first time, hence, the weekly run. We run the previous 36 hours (there is overlapping), and also once a week, we run it for the previous 7 days (we do get new trips during this time).

Let me know if you want me to send you anything. They are python scripts that use the URL feeds, and tokens. It also has token renewal for those that need it.

SaraAbasi
Occasional Contributor

Thanks Mariela, 

I appreciate you help and sharing the scripts.

I don't think it is possible to sent you a private message in Geonet community so I sent you a request in Linkedin. 

0 Kudos