topic Re: Webhooks not triggering randomly in ArcGIS Survey123 Questions

Webhooks not triggering randomly

PaulSweeney3 — Tue, 16 Mar 2021 20:49:48 GMT

We use survey123 and integromat extensively and it seems generally very reliable however recently I have users mention they have a small percentage of forms submitted, with a webhook attached , but no record of it on integromat. The record is successfully submitted to Survey123 but it did not trigger in integromat. I am unsure as to who I should raise a support ticket with. Is it more likely to be an integromat issue or survey123?

Re: Webhooks not triggering randomly

Andrew-Bowne — Wed, 17 Mar 2021 15:39:35 GMT

Have you checked the History of the Scenario to see if it was received but maybe errored? Or, maybe they will show in "Incomplete Executions"? Secondly, is it possible that the scenario was not turned on?

Re: Webhooks not triggering randomly

PaulSweeney3 — Wed, 17 Mar 2021 15:52:35 GMT

@Andrew-Bowne wrote:
Have you checked the History of the Scenario to see if it was received but maybe errored? Or, maybe they will show in "Incomplete Executions"? Secondly, is it possible that the scenario was not turned on?

Hi @Andrew-Bowne

Thanks, but no it's not as simple as that I wish. This is definitely missing executions with no history on integromat , on forms with Webhooks that have successfully submitted.

Regards

Re: Webhooks not triggering randomly

Andrew-Bowne — Wed, 17 Mar 2021 15:55:04 GMT

Are you using the free subscription or a paid subscription? I know that there are some limitations when you use free.

We have been using Integromat for a little over a year now and have been very happy. Right now we are running 7 Scenarios (some are very complex) and haven't had any issues (fingers crossed).

Re: Webhooks not triggering randomly

PaulSweeney3 — Wed, 17 Mar 2021 17:43:42 GMT

Yes we have a good few scenarios running on paid subscription and as i say its normally very reliable however the ones that are missed do cause headaches for users and confusion at times. Would be good to figure out if there is a specific reason.

Re: Webhooks not triggering randomly

Andrew-Bowne — Thu, 18 Mar 2021 13:06:27 GMT

I don't know if it exists but some logging would be really nice. Like, "Log requests sent to webhook". Then at least we could see outbound traffic. If it's seen going out, then the problem might be on the Integromat end. Webhook Logging might be a good feature request to add to Survey123 Ideas

Re: Webhooks not triggering randomly

KeithOlsen — Wed, 19 May 2021 16:41:12 GMT

I had this happen to me as well. A scenario that has been on and working for a year just randomly didn't pickup a survey payload that showed up on the Enterprise server (10.8.1). No trace of the survey in Integromat, and I searched the Portal logs. The next survey submitted worked fine. For some reason it doesn't seem to track the webhook triggers, which seems strange. Does anyone know where webhook triggers are logged? I'm using the survey123 3.12.230 on this one.

Re: Webhooks not triggering randomly

Andrew-Bowne — Wed, 19 May 2021 16:54:11 GMT

I just created an Idea for this. I suggest adding a comment and/or Kudo which will show Esri there is interest in having logging for webhooks.

Re: Webhooks not triggering randomly

KelliW — Thu, 20 May 2021 16:15:41 GMT

I just had this exact thing happen to me the past couple of days. I can't figure out why it wasn't sent through my scenario, there is no trace of it in Integromat but it is in Survey123. I wonder if it is a recent bug.

Re: Webhooks not triggering randomly

KeithOlsen — Thu, 20 May 2021 18:21:41 GMT

Could this be a recent bug @JamesTedrick ? It's strange that there doesn't seem to be any way to troubleshoot it.

Re: Webhooks not triggering randomly

TylerBragg2 — Mon, 14 Jun 2021 18:19:43 GMT

I've been having this happen to me as well, and it seems to be more often in the last few months. I have a paid subscription to Integromat and have 28 active scenarios running at all times, some of them being somewhat complex, some being fairly simple. I actually had this happen last week and again today on a scenario that has been running for about a year now with very few problems. The user reported the problem to me but I can't seem to nail down the source, except that it must be in the webhook. I checked the history of the scenario in Integromat and there are no errors, and more importantly, no record of the trigger at all. I've tried to look on the Survey123 side and I can't seem to find a way to validate that the webhook trigger was sent to Integromat. Then the next submission of the survey will work perfectly.

Re: Webhooks not triggering randomly

PaulSweeney3 — Mon, 14 Jun 2021 19:35:16 GMT

I think it's fair to say there is an issue so we re not all imagining it. Id imagine given the scale and number of requests its likely from time to time a webhook would fail . It would be great if someone could just confirm that , at least then we could work with it. As a precaution on some of my more important scenarios I have started creating round up scenarios that basically search the layer and process everything in the day again. It's extra executions but it gives piece of mind . I appreciate its not ideal for all scenarios but it does work well polling layers is defo more reliable if it suits the work flow.

Re: Webhooks not triggering randomly

PaulSweeney3 — Mon, 14 Jun 2021 19:44:38 GMT

Also what I intend to do is set up a Scenario in power automate and add the webhook to one of my existing forms in an integromat scenario. That way I can test if both Webhooks trigger or fail. My gut feeling it's the s123 app or an operating system issue due to connectivity I.e poor mobile data service

Re: Webhooks not triggering randomly

TylerBragg2 — Mon, 14 Jun 2021 20:42:14 GMT

That's not a bad idea at all. But, I'm thinking we could rule out connectivity issues as the cause because the webhook is not initiated at the mobile level; rather, it should be issued by ArcGIS Online at the feature service level(at least in my case, unless you are using Server, then from there). That's where a log file from AGOL would be nice so you can compare the submit time/date of the feature and the log file of webhooks to see if it skipped one. I'm thinking that's what's happening. But what you mention would be a good way to validate that. Or, I suppose there could be a blip with Integromat where it was NOT listening for a certain time period. Then AGOL could have sent the webhook but Integromat never saw it because it wasn't listening. In that scenario, Integromat wouldn't show it in the scenario history because it truly didn't receive it, even if it was because it "wasn't listening."

Re: Webhooks not triggering randomly

KeithOlsen — Mon, 14 Jun 2021 20:46:30 GMT

What Integromat function searches existing records on the featureclass? I've only used Integromat to respond to webhook triggers from Survey123. Would you be using something else to generate a webhook on a set schedule?

Re: Webhooks not triggering randomly

PaulSweeney3 — Mon, 14 Jun 2021 22:01:07 GMT

Hi @TylerBragg2

Really is that something you know for definite. I was always under the impression the app submits the webhook once it recieved a response from the server. That's why you had all the additional info on the user, survey information etc. I know you can generate webhooks from the feature service layer but I thought this was a separate process. I will admit I am no expert on this stuff so happy to be wrong on it.

Re: Webhooks not triggering randomly

PaulSweeney3 — Mon, 14 Jun 2021 22:05:53 GMT

The Http module along with the REST API Can be used to query a feature service with a typical query eg where objectid > 100 and return the json which you can parse and use in more modules in your scenario. You can schedule it to run at certain intervals. I think the intervals depend on your subscription. It's pretty useful once you pick it up.

Re: Webhooks not triggering randomly

KeithOlsen — Mon, 14 Jun 2021 22:40:01 GMT

Thanks! I'll check it out. --Keith

Re: Webhooks not triggering randomly

Andrew-Bowne — Tue, 15 Jun 2021 13:01:00 GMT

@TylerBragg2 wrote:
That's not a bad idea at all. But, I'm thinking we could rule out connectivity issues as the cause because the webhook is not initiated at the mobile level; rather, it should be issued by ArcGIS Online at the feature service level(at least in my case, unless you are using Server, then from there). That's where a log file from AGOL would be nice so you can compare the submit time/date of the feature and the log file of webhooks to see if it skipped one. I'm thinking that's what's happening. But what you mention would be a good way to validate that. Or, I suppose there could be a blip with Integromat where it was NOT listening for a certain time period. Then AGOL could have sent the webhook but Integromat never saw it because it wasn't listening. In that scenario, Integromat wouldn't show it in the scenario history because it truly didn't receive it, even if it was because it "wasn't listening."

I'm not necessarily sure that is true. I am using Portal ver. 10.6.1 with federated Enterprise 10.6.1. webhooks weren't supported with Enterprise until 10.7.

I believe that Survey123 is where the webhooks are administered and brokered to Enterprise (or AGOL). I am in agreement with @PaulSweeney3 . S123 is the initiator of the webhook. I think the big question here is - where would logging exist? If Integromat doesn't get the payload, it'll never know that it's being initiated. Since the webhook is presumably not initiated by Enterprise/AGOL then it really has no knowledge of the execution either. In my opinion the logging would have to exist on the S123 Admin level(https://survey123.arcgis.com/) where the initiation would have to be logged and a callback result would have to be logged as well. What I am not seeing is any sort of logging that currently exists for anything. Perhaps we could get some further insight from the Esri folks on this... @JamesTedrick @IsmaelChivite @Anonymous User

Re: Webhooks not triggering randomly

JamesTedrick — Tue, 15 Jun 2021 20:18:03 GMT

Hi @PaulSweeney3 @TylerBragg2 @KeithOlsen ,

We have observed scenarios where the webhook for a submission did not trigger in cases where the form included a number of attachments (images) and either a callback was specified to open another application or the user put Survey123 in background manually. We believe that this has been addressed for the next release of Survey123 - I would encourage you to get the current beta at the Early Adopter Community and test to see if the issue is resolved.