Survey123 Webhook not always firing

Question

Issue: It seems that periodically (on approximately 1% of our surveys submitted) we have Survey123 webhooks that are not firing when new surveys are submitted.

Setup: Our surveys (around 20 different surveys for inspection activities) are submitted to our ArcGIS Enterprise (10.9) and we have webhooks (for each survey) set up via the Survey123 website connected to our ArcGIS Enterprise.  The webhooks fire when new surveys are submitted and do not pass any event data.  The webhooks call FME server workbenches that ETL data and perform business logic.

Scale:  Our users submit 300-400 new surveys a week and about 2-3x per week the webhook does not fire.  This is a rare occurrence, and unpredictable.  Our evidence that the webhook does not fire is that the FME Server workbench is not called.

Question:  Is there any log that we can refer to in our ArcGIS Enterprise so that we can determine if the webhooks are firing??

Thanks,

- Jake

JakeMatthys · Answer

Thanks for the response Zach.  Our 'issue' with webhooks not firing as expected has diminished since I wrote this post so the issue no longer has an elevated priority.  I do appreciate your response to our inquiry.

- Jake

StrathconaBTS · Answer

We have inconsistent webhook behaviour in our surveys both on the web form and the mobile app. It turns out the webhook is fired directly from the app and web browser, NOT portal/arcgis.com once the data is received. This is an unfortunate implementation as it is highly susceptible to whatever the firewall settings are for the network the user is on. If the network blocks access to integromat.com (possibly make.com too) then the webhook will time out and never fire. We are currently solving this issue at our company as we have issues at most of our offices.

ZacharySutherby · Answer

Hello @JakeMatthys,

The way webhooks run in ArcGIS Survey123 is after Survey123 sends the applyEdits request to the feature service it then sends a request to the webhooks payload URL. This request to the webhook payload URL would be independent from your ArcGIS Enterprise organization and wouldn't show up in any ArcGIS Enterprise logging.

Depending on who your webhook provider is I would suggest checking out the logs for the webhook and see if either you are not seeing any run history for those specific records, or if there was an error in webhook execution. Both Integromat and Power Automate have run history logging and would be a good place to start.

If there are no run history for those specific records that could potentially indicate that Survey123 failed to send a request to the payload URL and would require more troubleshooting. If that seems to be the case I would suggest reaching out to Esri Technical Support and they will be able to assist with troubleshooting.

It would be good to know:

- If there is a pattern in the records that seem to not trigger the request?
   - Is it the same user sending the records?

- Is it around the same time?

- Is there some consistency between the data in the records?

- Is it specific to a certain device?

-  Are the records being submitted in remote areas with limited cell service?

- If your ArcGIS Enterprise environment is behind a firewall are your field users using a VPN to connect?

- If so the request to the webhook should be made through the VPN, is it possible to obtain the web traffic from the VPN?