Upsert Vs ApplyEdits

SimonJackson — Sat, 11 Dec 2021 23:14:09 GMT

Polling a GTFS-Realtime feed. Have some logic in Node to decode the Protocol Buffer and manually massage into a format suitable for pushing into an ArcGIS Online Hosted feature service.

Two example messages from my source back-end shown below. (typically around 3000-4000 messages in each packet)
Created logic to push this into an destination ArcGIS Online Hosted Feature Service with same schema, currently using ApplyEdits.
This push would happen every 30 seconds - (we need to be as efficient as possible)
We only require the current location of each unique transit vehicle in the hosted feature service
- i.e. We do not care about its previous history/breadcrumb trail.

We are trying to push the real time vehicle locations from the GTFS-Realtime feed into a hosted feature service.

Problem is, we keep creating duplicate points, when we need to:

Create new feature if the vehicle_id does not exist in destination feature service
Update existing features if the vehicle_id does exist

I was initially looking at ApplyEdits, but then stumbled across Append Upsert capability on the root REST endpoint for a feature service.

But, struggling to get it to work.

The unique ID is going to be the vehicle_id on both sides.

Problem: Getting duplicate records every time we push the JSON using ApplyEdits.

The Append Upsert endpoint sounded like it was a better operation to use:

"Upsert is used to add new features and updating existing features at the same time. It is a high performance API capable of handling large volume of data."

"The destination objectId or globalId fields do not need to be mapped to source fields of the same type. Source featureId can come from different field type as long as the values are acceptable destination featureId value."

However, we have tried a few different approacheswe are struggling to get this to work.

It sounds like we can push our response to a hosted feature service and use our source ID of

vehicle_id (string), and potentially push this into a GlobalID (string)? We have a GlobalID on the destination feature service.

What would the params looks like for an Append Upsert with a foreign ID on the source?

A copy of the test service is here with the Append operation available and fully open:

https://services9.arcgis.com/M6PRFWHKXSvV72Qg/ArcGIS/rest/services/Transit_Vehicle_Locations/FeatureServer/append

{
     "adds": [
          {
               "geometry": {
                    "x": 151.20631408691407,
                    "y": -33.882198333740237
               },
               "attributes": {
                    "vehicle_id": "4205062-10795-DN-91",
                    "trip_id": "4205062-10795-DN-91",
                    "route_id": "IWLR-191",
                    "bearing": 20,
                    "start_time": null,
                    "start_date": null,
                    "speed": null,
                    "congestion_level": "UNKNOWN_CONGESTION_LEVEL",
                    "TransitType": "lightrail"
               }
          },
          {
               "geometry": {
                    "x": 151.19384765625,
                    "y": -33.8746337890625
               },
               "attributes": {
                    "vehicle_id": "4205063-10795-DN-90",
                    "trip_id": "4205063-10795-DN-90",
                    "route_id": "IWLR-191",
                    "bearing": 221.6636657714844,
                    "start_time": null,
                    "start_date": null,
                    "speed": null,
                    "congestion_level": "UNKNOWN_CONGESTION_LEVEL",
                    "TransitType": "lightrail"
               }
          }‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

We have also been pondering if we should be looking at leveraging ArcGIS-REST-JS? This does not appear to have an append function, but the updatefeatures approach might be a better approach to deal with responses, especially when we move this to using a secure hosted feature service.

Any tips or suggestions would be really appreciated.

This service will later be available for all to use via a view layer.

topic Upsert Vs ApplyEdits in ArcGIS JavaScript Maps SDK Questions

Upsert Vs ApplyEdits