Event-Driven Integrations with Data Interoperability
Webhooks. If you don't code you might think webhook integrations are scary or at least need extra middleware apart from ArcGIS. Wrong on both counts. All citizen integrators who can put together an FME workbench with Data Interoperability extension, and have the extension for both Pro 2.6 and Enterprise 10.8.1 can automate webhook-driven integrations between Survey123 surveys and anything else Data Interoperability can reach.
I'll outline the scenario in case you haven't looked into this automation pattern before. Survey123 is a form-driven data capture app that manages a feature service where each survey is a point, line or area feature with attribution you design. The forms have rich behavior and support web and mobile devices. There are unlimited possibilities for downstream processing of survey features. What this blog will cover is this data flow:
- A Survey123 survey submission embodies a transaction for its feature service (add, update)
- A webhook defined for the transaction type is triggered at each submission
- The webhook URL is an execute call for a Spatial ETL web tool (geoprocessing service)
- The web tool performs an integration using the survey submission feature(s)
Putting this together is all about configuration, so lets get started. I'm going to describe the most common platform combination - Survey123 is hosted in ArcGIS Online and your integration server is an ArcGIS Enterprise portal - but if you have Survey123 on your enterprise infrastructure that will work too.
Because you will be creating a web tool that reads and writes a feature service you will need to share a web connection to your server as described in this blog, so your first step is to make sure the service owner account (by default named 'arcgis') on your server can start Workbench as described and use a web connection to where Survey123 feature services live. Don't proceed past this point until you have successfully shared a web tool that exercises the web connection, such as the test tool described in the blog.
I'm going to jump ahead now to explain how to avoid what might present as a silent failure of your webhook. Survey123 (i.e. https://survey123.arcgis.com) will be calling your server with a POST request, so it needs to be trusted. Below is a screen grab of my browser while I submit a survey, with the debugger pane open (Firefox). I have highlighted the POST request sent to my server - it worked. If your server rejects a request from Survey123 you'll see error messages talking about CORS - cross origin resource sharing.
See this topic about how to allow Survey123 to access your server - Cross Origin Resource Sharing.
Here is how it looks in the portal settings:
For me this setting wasn't enough, I also had to change HTTP response header settings in IIS. On your server, open IIS at the HTTP Response Header control, double click to open it and Add what you see below.
Now the server is set up lets get to configuring the data flow. If you have sharp eyes you'll see my test survey is about a caviar sandwich. If you research the coordinates its entirely possible you could actually buy one there! My real point though is Survey123 is sending a bunch of JSON, and it can be very complex for a big survey. If you take a look at other integration platforms you'll find that while they try and look like no-code approaches you might be forced to parse JSON in mid stream to get the data you want. However, Survey123 is always writing to a feature service, which of course Data Interoperability natively understands, so we have the luxury of forgetting about the JSON flying around and going directly to the feature service. Fantastic!
First though, create your survey. Here is mine:
When a survey is submitted a feature is written to its feature service and then any configured webhook is triggered - you can see the applyEdits POST in my browser debug view above, ahead of the webhook POST. To support the integration geoprocessing service finding new records, my survey contains a hidden, required question 'Integrated' with a default value of 'N', which my integration edits to 'Y'. I created my survey in the web designer, which doesn't support hidden questions, then I edited it in Survey123 Connect to add the question, here is how it looks when edited:
And when saved to the survey:
You have to update and republish the survey to apply the schema change.
So now my survey has a field I can use as a processing selection flag. I created a couple of records so I could design the integration workspace, and here it is:
Its also in the blog download. It doesn't do any actual integration - that's your job - but you can see the pattern. Read records where integrate = 'N', do your stuff, then update integrate to 'Y' and write back to the service with GlobalID as the key field. Make sure your tool source is embedded.
If you have a large number of collaborators in your survey and by chance two surveys are submitted simultaneously it doesn't matter if one execution processes two surveys and another none. If you're worried about latency of surveys writing to the feature service then begin your process with a Creator-Decelerator-FeatureReader combination.
I manually tested the processing before publishing a web tool by submitting surveys and running the tool in edit mode. At that point I'm ready to share the web tool. Run the tool from the toolbox to create a History item then share it. You'll notice the service is to be public, if that can't be done at your shop then take Survey123 onto your enterprise infrastructure.
Make the tool synchronous for a start. If you find your available instances can't keep up move to asynchronous.
My tool has no parameters.
When it completes sharing, go to the service URL and copy the value, here is mine:
So my webhook URI is: https://dev0014223.esri.com/server/rest/services/WebhookIntegration/GPServer/WebhookIntegration/exec...
Now all that remains to do is configure the webhook in my survey.
Now when a sandwich survey comes in my webhook integration triggers and Integrated = 'Y'.
Genuine no-code integration, courtesy of ArcGIS Data Interoperability. If you see me at local sandwich stores I'm working - honest.