8/15/24: Update and quick fix for issues with Webhooks not running in the ArcGIS Connector

[AnthonyLatini]

There have been multiple reports here on the forums and Support issues logged for a recent problem with webhooks not working. We have been searching for the root cause of this issue over the last two weeks and have found the problem as well as a quick solution.

In short, a change was made in the underlying ArcGIS Online system which inadvertently affected webhooks not just in Power Automate, but in other client applications as well. This change made it so that 202 responses were no longer accepted in the webhook API and only accepted 200 responses.

If you had recently created a webhook, it might have appeared working about 5 to 6 times before it turned off. The results on the admin page for your feature layer might have looked something like this:

Once the webhook reached its threshold of unsuccessful events, it would turn the webhook status to false. This would then stop your flows from triggering and running subsequent actions.

Currently, a patch is being worked on to resolve this issue in the ArcGIS Online system. Please note that we do not have an ETA on when the patch will be pushed, but we will monitor this closely.

The fix has been deployed to production by the ArcGIS Online team and should be live. You may need to enable your webhooks again with the steps below to see the update. 

However, we have a quick solution to get your flows up and running again. You only need to edit your flows once even after the patch has been deployed. Please see the steps below.

Thank you for your patience on this issue as we work through it!

----

1. In Power Automate, navigate to your existing webhook flow
2. Open the flow and click “Edit”
   
3. Add a new action right after the trigger
4. Search for Request in the action search box
5. Select “Response” from the search results

1. In the Response action, the “Status code” should be defaulted to “200”
2. Save the flow
3. Navigate to your feature layer in your organization
4. Scroll to the bottom of the page and open the service URL

1. In the upper right corner, click on Admin

1. At the bottom of the page, click on “Web Hooks”

1. Next, select “Activate all” to ensure all of the webhooks are set to True

   Click on "Activate all" to enable all webhooks. 

   
2. The flows in Power Automate should now start to run again

[Scott_Sambell]

Thanks @AnthonyLatini . What about FME?  We don't use Power Automate and this bug is giving us major headaches trying to keep eveything running. 

CC @EnricShen @WillyG 

[NigelCharman]

Thanks for the workaround.

We also needed to Publish our Power Automate flow (since it's packaged as a Solution).

I assume we also need to push the Activate All button below after clicking on the Activate All link:

Nigel

[AnthonyLatini]

@NigelCharman - Ah thank you for that! I will edit my post. I think it saved an earlier draft. Good catch! 😀

[AnthonyLatini]

@Scott_Sambell - Hi there. I am not too familiar with FME. Can you post a screenshot in a private message to me so I can take a look? 

[RiainGarcia]

Following

[HackRentmeesters]

We're also using Webhooks in FME Flow which this issue makes our workspace unfuctional. Can you provide a solution for FME as well?

[AriEpstein]

I'll add that our organization is working around this by switch to a pull approach that avoids webhooks altogether. We created a scheduled flow that uses the Get data from feature layer action to pull features that were created/modified since the last run. Compared with webhooks, this has been simpler to implement for our use case and more robust (we'll get immediate reporting through the flow if the pull fails whereas the webhooks tend to fail in less detectable ways).

[Erwinvan_Veen1]

Thanks for the suggestion. Quick question: what am I supposed to do when I cannot find the Request nor the Response actions?

[ahargreaves_FW]

@AriEpstein can you provide a screenshot or example of this flow? How do you determine what's new since the last time the flow ran? It is simply something like:

1. Flow runs every hour
2. Each run looks for new features since: now() - 1 hr?

It sounds much better and more stable...

Thanks

[Bryan_Wade]

Would this work around work for flows use Survey123? That is where I am having the issue with the connection being dropped in Power Automate?

[AbdiHassan]

I ended up creating an FME workbench that activates all my webhooks every minute which is working in the interim. You need to use the HTTP Caller to grab the webhooks and activate them, I can send the FME script if needed

Thanks

[Erwinvan_Veen1]

@ahargreaves_FW Although we need to test it more, in PowerAutomate I solved this as follows. Without web hooks.

A. Recurrence 1 minute.

B. Get data from feature layer using where clause. The where clause is:
GEWIJZIGD_OP >= CURRENT_TIMESTAMP - INTERVAL '1' MINUTE

Please note that GEWIJZIGD_OP is my editor-tracking field for the last edited date. This should be changed to your situation. Perhaps an entirely different field suits your needs better.

Hope this helps.

[CarrieKeneally1]

Hi Any word on when this might be fixed?  We are having weekly issues with our notifications going out to our various departments.  This a huge issue for our departments and being notified about issues that are sent in by the members of the public.  

[AnthonyLatini]

@CarrieKeneally1 - There has been an update from the ArcGIS Online team on this issue. They have pushed out a patch to production and should be live. I have tested this on my end and the issue appears to have been resolved. You may need to reenable your webhooks to see the changes appear. The steps I have listed out in the initial post of this thread about "activating" the webhooks again should solve it. 

Let me know if you have any issues getting these up and running again! 

[Tiff]

Hi @AnthonyLatini thank you for the update. Just to confirm, has the webhook deactivation issue been fixed, or is the workaround posed in this post still required? 

[BrianFischer]

Were still experiencing this.  We documented our webhooks got deactivated sometime between Oct. 16 9pm central and Oct. 17th 10am central by something that was not us.  Anyone else have the same problem last night?