Bike-Share Feeds with GeoEvent & Koop
Let’s pretend a request has come in to build an Operations Dashboard with all the bike-share feeds around Washington, DC. We are told this task will be easy because all these feeds adhere to the General Bikeshare Feed Specification (GBFS) and our prototype must only meet the minimum of the GBFS specification; i.e. where is the station and how many bikes are available now. Our manager informs us that our solution should use ArcGIS GeoEvent Server (GeoEvent) to push these real-time bike station updates into a Feature Service published with ArcGIS Server. With our goal established and our tools selected, we can get started on the prototype.
Getting the first draft stood up with GeoEvent doesn’t take any time at all. We build a GeoEvent Definition (GE Definition) by importing the schema from a Feature Service that we published after reading through the field definitions in the GBFS spec. Knowing that the GBFS specification breaks up the station information and the station status into 2 separate feeds, we duplicate the GE Definition and pare each down to the required fields. From there, we make 2 Input Connectors (Poll an External Website for JSON) for each bike-share provider we are interested in mapping.
In the image above, GBFS Info and GBFS Status (probably should have been labeled “Outgoing” for consistency) were both copied and edited from gbfs_sde_GBFS (our Feature Service), while the two ending with “Incoming” were automatically generated when creating the Input Connectors. The beauty of using GeoEvent for a scenario where all the inputs will have a reliable schema is that we should only need these 4 definitions to facilitate an arbitrary number of inputs/providers. After creating an Output Connector (Update a Feature) that will move incoming values into our Feature Service, all that is left to do is publish a GeoEvent Service with 2 Processors (Field Mapper) that will map the bike-share information and status feeds to their corresponding fields in the output.
Shown above is the baseline we need in GeoEvent to build an Operation Dashboard that will display real-time updates to our bike-share feeds. If our manager comes through the door with another provider (let’s say Jump), all we do is create 2 more Input Connectors, ensure they are pointed at the correct Field Mapper, and then update our GeoEvent Service (shown below).
What do we like about this solution with GeoEvent? It was easy to build, and it will be easy to add new bike-share providers. We can even leverage the GeoEvent REST API to automate the process of building out the remaining providers with Python. However, adding 2 Input Connectors for each new bike-share means that the GeoEvent Manager quickly becomes cluttered and difficult to manage. What if we were asked to create a dashboard of bike-share feeds from around the world? Another pain is thinking about expanding the schema past the baseline needed to get approval from our manager. We’d likely need to publish another service, reimport the definition, edit that definition, and then ensure the field mappers were handling these new inputs. And that doesn’t even address the problem of many bike-share feeds offer a range of information beyond the baseline that does not necessarily match between providers. Finally, what if we didn’t have a license for GeoEvent and ArcGIS Pro? Even if we did have permission to use these tools, we might have to spend the time setting up GeoEvent/Enterprise in the first place (this is far less troublesome than some would have you believe).
But the dashboard is done and since we knocked out our GeoEvent pipeline so quickly, there is some extra time to see how Koop can help us deliver on the same request. What is Koop? The GitHub repository says that Koop “exposes a Node.js server that can translate GeoJSON into the Geoservices specification supported by the ArcGIS family of products”. For us, this means that a successful Koop provider only requires you to write enough JavaScript to convert the bike-share feeds into a GeoJSON object. There is more functionality in the Koop ecosystem (such as the ability to cache results instead of simply doing a translation), but this prototype just needs to get in front our manager quickly. After a cursory read through the Provider specification in the Koop documentation, we pull down the sample and start hacking.
For simplicity, we will define the information for the bike-share feed in the default.json file that ships with the provider sample; this allows us to write less code and use what is already inside the provider sample for model.js. Just assign a number (starting at zero) to each pair of station URLs and include some metadata information to help clients consuming our final Geoservice.
This JSON file handles the information that would have been specified in the Input Connectors we needed with the GeoEvent solution. With the bike-share feeds stored somewhere, we can start working on the core of this Koop application: model.js.
The 40 lines of JavaScript shown above are accomplishing what we outlined in our approach with GeoEvent. All we are doing is grabbing the layer id from the request (e.g. 0 which is for the Capital Bike Share), fetching the JSON response from both endpoints in the configuration file, and then creating a feature for each station. However, there is something powerful about how the properties for each GeoJSON feature are being built and this can be seen at line 24 (properties: s). Since we are not relying on a published service with a fixed schema, we can append whatever properties we want to the outgoing feature. This mean our logic within Koop can remain unconcerned about the details of a given provider and that is one of the huge benefits of this approach. There is also a bit more flexibility in how we manipulate the data before it goes back to the client since we are not restricted not to the out-of-the-box Filters/Processors in GeoEvent, but simply to whatever we can do with JavaScript.
So, what do we like about Koop over GeoEvent? For starters, it is completely free. This lack of a price tag is balanced by the fact that you will need to figure out how to expose/secure this service for consumption. Aside from sitting it behind Apache and getting a certificate with Let’s Encrypt, you might also have to think about authentication which is a completely different story (though I have heard of people getting along just fine with middleware to handle PKI authentication). Another benefit to the Koop solution is that you do not need to understand all the new terminology that goes along with GeoEvent, so this is something that can be stood up by someone who is not familiar with GIS or the ArcGIS platform. Additionally, moving this solution between environments or sharing it to people would be trivial when compared to GeoEvent. But perhaps the biggest reason to experiment with Koop (if you are a developer) is the ability to arbitrarily extend Koop into solution that does more than just return a Geoservice that can be consumed by ArcGIS clients.
This aim of this post was not to convince you that Koop should be used over GeoEvent. Our manager is this fictional example probably wants us to use GeoEvent because it is integrated tightly with Enterprise and ensures our focus is on moving data and not on custom development. However, now you know that there is a free, flexible, and easy-to-extend Esri product that your developers can put in their toolbox for whenever you hit a ceiling with out-of-the-box functionality, or when you have time to think about creative solutions.
You can find the (embarrassingly simple) code I wrote for this blog at the following GitHub repository.