Sneak Peek: Survey123 and Webhooks (Integromat)

22380
60
03-05-2018 12:44 AM
IsmaelChivite
Esri Notable Contributor
31 60 22.4K

----------------------------------------------------Note start------------------------------------------------------------------------------------------

 The original content of this post is obsolete. All functionality described is now generally available as described in this video:

----------------------------------------------------Note end------------------------------------------------------------------------------------------

We are happy to announce that Survey123 will introduce support for Webhooks shortly.  If you are not familiar with webhooks, think that you will be able to integrate Survey123 workflows with Micosoft 365, Google Sheets, Box, Evernote, GMail, SMS (Simple Message System), Apple Push Notifications, Salesforce and hundreds of cloud services more!

What is a Webhook? The concept of a Webhook is simple. A WebHook is an HTTP/s callback: an HTTP/s POST that occurs when something happens; a simple event-notification via HTTP/s POST. 

The beauty of webhooks is that they are a commonly accepted pattern by many applications out there, so a whole ecosystem of different providers allows you to work with webhooks and automate processes involving things such as Microsoft 365, Evernote, Google Sheets, SalesForce...and now Survey123.

What can I do with Webhooks and Survey123? As you will find soon, many different things, but I will give you a simple scenario: Imagine you want to send an email to someone when a high priority incident is reported through a particular Survey123 form. Imagine too, that if the incident is low priority then you want the incident to be logged in a Google Sheets spreadsheet.    By configuring a webhook with Survey123, you can trigger a flow that orchestrates Gmail and Google Sheets to do what I just described. The flow could look something like this:

The flow above was built using Integromat. Integromat is a cloud application that helps you visually orchestrate different cloud services.  Integromat supports initiating these flows, or scenarios as they call them, with a webhook triggered by Survey123.

How can I do this myself? I am slow typing, so I recorded a step by step video so you can get a hang of it.  You will be able to do this by using the Early Adopter version of Survey123

Do I need to use Integromat? Not necessarily. Integromat is one of many possible solutions for using webhooks and Survey123. Other solutions include Zapier, Microsoft Flow, Tray.io, IFTT etc.  You can in fact trigger a webhook that calls your own custom developed web service, an ArcGIS geoprocessing service, an FME service etc... 

Does Esri plan to have connectors for Zapier, Microsoft Flow etc? Very likely, but we are still working out what is the specific set of connectors we will create. We want to use this Early Adopter Program to figure out which connectors you would like. For now, you can use the generic webhook connector! 

To what Survey123 event can I associate a webhook? For now, only to the submit operation in a form (Web Form or Survey123 field app). If you have use cases for other triggers, let us know.

Updated June 17, 2018: Are there any restrictions to the use of webhooks using Survey123? Yes: Webhooks will only be triggered if the end-user is logged in into the Survey123 field app or Survey123 Web Form. Survey123 webhooks will not be triggered if the form is shared publicly and the user submitting the form is not logged-in.

When is this functionality planned to be made available? We are targeting version 3.0, which will release around July 2018. Our Survey123 Early Adopter Program now lets you experiment with this feature. We are looking for your feedback through the Early Adopter Program!

60 Comments
AmyWork3
New Contributor III

This is great to see and I am excited to implement it in a few surveys! One question I have for future releases: Are there plans to make webhooks be available for public forms? This could be extremely valuable when you have created a survey to collect data generated by the public and you are interested to know when certain categories are submitted. 

IsmaelChivite
Esri Notable Contributor

Thanks Amy! Our initial thinking is that we will enable Webhooks for private surveys only. Having said this, I would like to learn more about your specific use case. I will email you for details.

by Anonymous User
Not applicable

This looks great

But, I too would like this feature to be enabled for public forms.

THANKS

LauriDafner
Esri Contributor

This is amazing, very excited!

HaleyNelson
New Contributor II

This is very exciting! Thank you!!

I, too, would be extremely interested in utilizing webhooks for public forms. I'd be happy to discuss our workflows with you.

DougEades
New Contributor

Hi Ismael - LOVE, LOVE, LOVE this feature and I agree with the prior comments regarding use on public forms.  A very real use case scenario would be where a link is provided for community residents to complete a form and the results would be e-mailed to a specific monitor.

Also - from an Emergency Management point of view - a "generic" damage reporting survey link could be provided to residents where they would self report their damages.  If a specific trigger is met (say "house is destroyed"), then an e-mail would be sent to the local Emergency Manager.

I've downloaded Survey 123 Connect Beta and am building this out now.

Thanks!

Lake_Worth_BeachAdmin
Occasional Contributor III

this is great!!!!

Surprised ESRI is pushing workflows to 3rd parties though...

DanielMackey1
New Contributor

+1 for webhooks in public forms. I have a client who would like to publish a fauna sightings survey to the public - when certain conditions in the survey are met in the submission, they would like to alert a staff member via email to perform validation.

DeonLengton
Esri Contributor

Quick question - what is the URL for the Survey123 Early Adopter website?

HélèneTouyéras
Occasional Contributor II
AdamInglis1
Occasional Contributor

Are we able to add the Report for the submission as an attachment?

JamesTedrick
Esri Esteemed Contributor

Hi Adam,

Currently, we do not support the generation of a report to be triggered as part of the web hook process.  This is something we are considering for implementation at a later date.

by Anonymous User
Not applicable

Hi ichivite-esristaffJTedrick-esristaff

Is there a way to include attachments and repeats in emails?

Thanks,

Gee

Lake_Worth_BeachAdmin
Occasional Contributor III

will public forms be enabled for this new feature in the July 3.0 release?

Greatly desired for public forms

AlexP_
by
Occasional Contributor III

Hello,

Would these trigger to web apps?? Such as crowdsource manager/reporter?? Please confirm. Thank you!

JamesTedrick
Esri Esteemed Contributor

Hi Alex,

Web hooks trigger the back end services that other web apps use.  In the case of crowdsource manager/reporter, a web hook should not be necessary - those apps can have the Survey123 layer in their maps directly.

AlexP_
by
Occasional Contributor III

I am not sure If I am understand correctly. Is this support only coming from Survey123? I have a project coming up but it is not coming from Survey123.  It is coming from Crowdsource reporter using feature service layer.  

JamesTedrick
Esri Esteemed Contributor

Hi Alex,

Currently this will be a Survey123-only feature. 

deleted-user-w7xjgPZ_tb8a
New Contributor

Hi Ismael/James,

Is there a reason why it's not available for public forms? I am keen to test this in few of our public survey forms. The one, in particular, is the wildlife roadkill survey in Greater Sydney region, where the public can use the survey 123 form to notify the incident.

IsmaelChivite
Esri Notable Contributor

Hi. We are evaluating the security implications of using webhooks in public surveys.

Jean-YvesLandry1
Occasional Contributor

  I was thinking of using a word template, fill in using the the output from the web hooks, then send it as a report through email.  I however don't see the attachments (imagery) as an output in the webhooks. Is this possible?

Lake_Worth_BeachAdmin
Occasional Contributor III

yes join the early adopters community and read through some of the posts there is a link for a how0to send attachments via emails

MalcolmJ
Occasional Contributor II

Our main use of webhooks would probably be on public surveys so we'd be very keen to see this expanded to cover them as well

IsmaelChivite
Esri Notable Contributor

Malcolm Jacobson‌  Paras ACHARYA‌  Alex P‌  Joe Head‌  Geethaka Fernando‌  Doug Eades‌  Haley Nelson‌  Amy Work‌  Daniel Mackey‌ 

Webhooks will be supported with public surveys at version 3.0 (July 2018)

DougEades
New Contributor

Wonderful news, Ismael Chivite!  Thank you so much for hearing our input on this - what a wonderful platform and step forward!

HaleyNelson
New Contributor II

Much appreciated!!! Thank you!

DeonLengton
Esri Contributor

Hi Ismael

Quick question - I have a suspicion that the Survey123 WebHook does not fire when the feature originates from the Inbox. It seems to only fire for new Survey records. Is this how it was designed or could it be a bug?

From the WebHook description though it looks like its only intended for newly captured records... 

Regards

Deon

MichaelKelly
Occasional Contributor III

Hi Deon,

This is a current limitation of Web Hooks via Survey123. Web Hooks will only trigger when new records are collected, not when edits are made via the Inbox. I believe this functionality is planned for a future release.

Mikie

by Anonymous User
Not applicable

Hi team,

 

I have noticed that webhooks fail to load in some instances.

 

I use Integromat for my webhook provider.

But, I think the issue happens at Survey123’s end (as I can’t see any errors or queues on Integromat’s end).

 

The most significant of these happened

yesterday - 5 September (between 11:02am and 12:09pm - Sydney, Australia).

We received, 5 surveys during that time and all of them failed to trigger webhooks. But, afterwards it was working fine.

 

Could you please look into this? Ismael ChiviteJohnathan Hasthorpe

 

Regards,

Gee

Lake_Worth_BeachAdmin
Occasional Contributor III

Geethaka Fernando‌ there was an Azure outage yesterday that disrupted many services on AGOL including Survye123.

- specifically disrupting the handling of feature services.

can you confirm the issue is still present currently?

by Anonymous User
Not applicable

Hi Joe Head‌, thanks for the response. Yes, it's working currently

But, this happens a bit too frequent for my liking.

Is there a way to modify this from Survey123's end, so if this happens it has the ability to fire up those webhooks again?

Gee

JamesTedrick
Esri Esteemed Contributor

Hi Gee,

Currently, Survey123 only supports webhooks on initial feature submission.  Other events, such as editing a feature are planned for the future.  If the survey was affected by the service interruption, then both the webhook should not have fired, but also the form should not be marked as sent (still in the Outbox) - submitting the form again should trigger the webhook on successful submission.

by Anonymous User
Not applicable

Sorry, I forgot to mention that in all these instances the actual surveys were submitted fine.

But, the webhooks didn't work.

MarioFederis1
New Contributor III

Excited to know the edits submission trigger is slated for future release.

As another suggestion, perhaps a granular trigger would be useful. Such as when certain fields as changed to specific values

JasonHinsen2
New Contributor

To what Survey123 event can I associate a webhook? For now, only to the submit operation in a form (Web Form or Survey123 field app). If you have use cases for other triggers, let us know.

 

Going along with the above comment in the body of this article and with Mario Federis‌, I had the idea of having hidden fields in the survey that reference a feature class in an enterprise geodatabase that serve as a trigger to send out notice of a deadline or expiration date.  If a customer has a need to visit a site and fill out a survey as part of a regularly scheduled event, this trigger could send out a notice to certain parties to remind them that this process cycle is going to start again soon.  For example, if policy dictates that assets be tested for adequacy every year and the asset record contains a maintenance date, the survey could hide this value from the survey, but trigger a webhook to generate emails to field workers and supervisors 60 days prior to expiration.  If this is somehow already possible or if this functionality will be added in future releases, please let me know.

KerrHewin
New Contributor II

Hello, 

is there any update on having webhooks sent when changes are made from the inbox?

JamesTedrick
Esri Esteemed Contributor

Hi Kerr,

Support for edit webhooks has not been implemented yet.

JamieTill1
New Contributor III

Hi there,

We have been testing and  using Integromat extensively with ArcGIS Enterprise. With the release of the updated 'Create Feature Report' we continually get an Invalid Token 498 message 

This was working about a month ago with no issue, now we can't get past this message. We have tried surveys with Hosted Feature Services, services coming directly from server. We have followed the instructions on the Early Adopter site, registered the app within Enterprise etc but still no further. Works perfectly with ArcGIS Online. As you can see above the connection to Enterprise is being made as I can select the survey, it is the error message after, not sure what it is expecting to see. 

We have a number of customers wanting to use this service, would be great to get this resolved, any help or guidance appreciated.

Regards

Jamie

JamesTedrick
Esri Esteemed Contributor

Hi Jaime,

In Integromat, can you click on 'Connections' and verify the Survey123 connections.  You should be able to reconnect any that have become disconnected.

JamieTill1
New Contributor III

Hi James, 

Thanks for the reply, I have verified the connection back to Enterprise, green tick comes up to say it is OK. I have tried creating a brand new one as well. Have gone through the app ID process following the steps but I keep getting the Invalid token message. I have had one of my customers also have the same issue. Maybe something I am missing?

Cheers

Jamie

JamesTedrick
Esri Esteemed Contributor

Hi Jamie,

It sounds like you've checked the connection properly.  That being said, I missed that you were specifically using the Feature Report beta functionality with ArcGIS Enterprise.  Unfortunately, we are deprecating support of the Feature Report functionality with ArcGIS Enterprise as we prepare to release the functionality from beta; the Integromat connector already reflects this.

JamieTill1
New Contributor III

Thanks for the update James. So will the Create Feature Report be available again in Integromat? If so is there a timeframe for the production release?

Cheers

Jamie

JamesTedrick
Esri Esteemed Contributor

Hi Jaime,

The issue isn't Integromat & the Feature Report functionality, rather ArcGIS Enterprise & the Feature Report functionality.  With the initial production release of Feature Report, ArcGIS Enterprise services will not be supported directly (they may be supported via proxied URLs).  We are considering ways to enable ArcGIS Enterprise to be able to use the Feature Report service in the future.

JamieTill1
New Contributor III

Thanks for the update James, appreciate it. We have a number of forestry customers that have Enterprise and a requirement for Word / PDF copies of their surveys, the Feature Report functionality is an ideal fit. If there is a way that we could use this via proxied URL's we would be really interested.

Regards

Jamie

JackPalmer
New Contributor II

Hi James,

What is the reasoning behind dropping the reporting functionality from ArcGIS Enterprise? If this is dropped completely, it is going to severely hinder our business processes. Creating word reports for our surveys is one of the major workflows to be able to show this data to third parties, often required by legislation. Is there going to be an option to get around this which members in our organisation will be able to do relatively easily?

IsmaelChivite
Esri Notable Contributor

Jack PalmerJamie Till‌  We are looking at ways to bring a solution for report generation in Survey123 to ArcGIS Enterprise users. Stay tuned.Thanks for your comments.

[Update] We brought back report generation for Enterprise users as described in our 3.5 release announcement: https://community.esri.com/groups/survey123/blog/2019/06/28/world-ufo-day-release-35 

Miguel-Fernandez
Occasional Contributor

James Tedrick ichivite-esristaff I'm hoping to implement Survey123 and Integromat as an email notification system. Can you confirm that when a survey is submitted and triggers the webhook to send three emails, that is considered three operations? Or is that considered four operations with the submission of the survey counted as an operation?

Thanks,

Miguel Fernandez

JamesTedrick
Esri Esteemed Contributor

Hi Miguel,

I'd suggest you review Counting the number of operations – Integromat Support, which describes the operation counting process.  In a very basic implementation of what you describe (watch a survey > e-mail 1 > e-mail 2 > e-mail 3), that would be a total of 4 operations for one survey submission. 

Miguel-Fernandez
Occasional Contributor

JTedrick-esristaff Thanks for the response and info!

by Anonymous User
Not applicable

Hi James,

I was wondering if Web-hooks works with Collector for ArcGIS. We use Collector in conjunction with ArcGIS enterprise/Portal to host feature services.

Thanks

PJ