topic Re: Feature Service Webhook in ArcGIS Enterprise Questions

Feature Service Webhook

Thomas_Fuhrmann — Fri, 22 Mar 2024 14:15:43 GMT

Hi all,

I'm trying to set up a Feature Service Webhook in ArcGIS Enterprise 11.1. The webhook should send a POST request to a custom receiver when a feature is created. Setting up the webhook is possible, but when I create a feature there isn't any request sent.

I tested with the same custom receiver and an ArcGIS Online webhook. In this situation the post request is sent correctly. I assume that it can't be related to CORS since it works in this situation and I also added the hostname of my ArcGIS Enterprise server as an allowed origin to my custom receiver's web server (Apache). And I'm able to send a post request to the custom receiver from the ArcGIS GIS server machine manually.

Port 443 is open in the ArcGIS Enterprise network. The whole ArcGIS Enterpise installation is behind a KEMP reverse proxy/load balancer.

Does anyone have any ideas?

Re: Feature Service Webhook

KevinHibma — Tue, 26 Mar 2024 14:42:29 GMT

I'd turn debug logs on in the server. Perform your feature service action (add feature), and then check the logs. You should see messages specific to webhooks. Based on your explanation it sounds like you will NOT see the webhook dispatched message. But you might see errors or other log messages that could be helpful.

Re: Feature Service Webhook

Thomas_Fuhrmann — Tue, 02 Apr 2024 11:28:34 GMT

Thanks for the reply @KevinHibma! Unfortunately there aren't any debug logs related to this service after performing the feature service action (create feature).

I found the following documentation which could be relevant in this case:

25672 and 44369 (TCP)—These ports must be available (free) on the relational data store machines because service webhooks require them.
45671 and 45672 (TCP)—These ports on the relational data store machines must be available to allow communication with the ArcGIS GIS Server sites because service webhooks require them.

I checked these ports and they aren't open yet.

Re: Feature Service Webhook

Thomas_Fuhrmann — Mon, 01 Jul 2024 09:31:01 GMT

@KevinHibma After checking our firewall settings, I tested feature service webhooks again. Unfortunately there is no post request sent after creating a new feature. I tested the same webhook receiver with organization webhooks and it works fine. The server logs do not show any error messages.

Do service webhooks require a running RabbitMQ service?

I checked under Windows services and could not find any running RabbitMQ processes.

Is there any configuration parameter in ArcGIS Server that allows me to enable webhooks?

Re: Feature Service Webhook

KevinHibma — Tue, 02 Jul 2024 18:06:27 GMT

Hi Thomas,

You won't see a "RabbitMQ" process by name. It's wrapped in another process. But I wouldn't go real deep looking here.

You said you're using ArcGIS Enterprise 11.1? When you check your Server debug logs, do you see any errors related to webhooks? If not, thats good. If so, can you paste them here? If you're getting errors I'd suspect you would be seeing them quite often.

If no errors, within the Server administrator, can you check /server/admin/system/webhookprocessors (you'll need to type the webhookprocessors part in, it's not a published API). If you check the config, is it populated? If not, can you restart you ArcGIS Data Store, and then your ArcGIS Server, then Portal (the restart is just to get everything into a state where that config would have information. Note - if it is already filled in, just try restarting ArcGIS Server.

Further to your question, no, there is no configuration parameter you need to set or modify to enable webhooks. The config I mention above would automatically be filled in by the system: you cannot manually update it.

Re: Feature Service Webhook

Thomas_Fuhrmann — Wed, 03 Jul 2024 14:16:55 GMT

Thanks for the informative and quick reply, @KevinHibma ! In the meantime we upgraded to version 11.3. I checked the config for the webhookprocessors and it's already populated. Our ArcGIS server automatically restarts once a week.

In ArcGISDatastore\etc\queuestore-config.json I found the following config. The datastore.release version should be 11.3, datastore.release.rabbitmq should be 3.13.0, datastore.release.rabbitmq.erlang should be 26.2 and datastore.lastknownstatus should be Started:

{
"datastore.release": "11.0.0",
"datastore.release.configstore": "1.5",
"datastore.release.rabbitmq": "3.9.13",
"datastore.release.rabbitmq.erlang": "24.2",
"datastore.name": "e3t13dt",
"dir.dbdata": "D:\\ArcGISDatastore\\rabbitmqdata",
"db.host": "GIS-DS01.GBA.GEOLBA.AC.AT",
"mode": "primaryStandby",
"replication.role": "PRIMARY",
"datastore.registered": true,
"localstore.lastknownstatus": "Upgrading",
"owningsys.type": "GISSERVER",
"healthcheck.enable": true,
"datastore.lastknownstatus": "Stopped"
}

I checked the logs of our Data Store machine and found the following entries related to webhooks:

<Msg time="2023-01-26T14:38:12,777" type="WARNING" code="110616" source="Data Store" process="8052" thread="17" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="" requestID="">The Webhooks database for the site "382a15d6-5c5e-44ba-89fc-c3bef95bd0d3" is not created and registered.</Msg>
<Msg time="2023-05-04T13:10:45,52" type="SEVERE" code="111902" source="Data Store" process="5300" thread="27" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="" requestID="">Error configuring the queue store. null</Msg>
<Msg time="2023-12-21T13:55:31,925" type="SEVERE" code="111902" source="Data Store" process="5896" thread="24" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="" requestID="">Error configuring the queue store. C:\ArcGIS\DataStore\framework\runtime\rabbitmq\erl-24.2\bin\erl.ini (The system cannot find the specified path)</Msg>
<Msg time="2024-06-12T14:04:50,77" type="SEVERE" code="111902" source="Data Store" process="8176" thread="18" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="0.0" requestID="">Error configuring the queue store. Cannot invoke "com.esri.arcgis.datastore.client.UpgradePrerequisiteCheckResult$StoreCheckResult.getUpgradable()" because "<local11>" is null</Msg>

Re: Feature Service Webhook

a-j-campbell — Mon, 21 Oct 2024 17:32:48 GMT

Hey @KevinHibma! I'm going to revive this thread in hopes you'll be able to help me out! I've been working on developing a custom webhook receiver using a synchronous GP service and believe I've got things correctly written to handle a Hosted Feature Service webhook (ArcGIS Enterprise 11.1) but am getting a generic error in the debug logs when I try to trigger the webhook.For some additional context, these are my Webhook Settings:

HookUrl	https://<server_url>/arcgis/rest/services/<folder>/<service>/GPServer/webhook_handler/execute
Active	true
Change Types	FeaturesUpdated
Payload Format	json
Content Type	application/x-www-form-urlencoded
Signature Key
Schedule Info	{"name":"","state":"enabled","startAt":1729524167012,"recurrenceInfo":{"interval":20,"frequency":"second"}}
Creation Time	1729524191497
Last Update Time	1729528085234

Based on this discussion, I'm seeing the "calling" message for this webhook:

Webhook log: Calling webhook url ==> https://<server_url>/arcgis/rest/services/<folder>/<service>/GPServer/webhook_handler/execute for GP Server.

But don't see the "dispatching" message. I do eventually get the following (without an Exception trace):

Webhook log: Calling webhook URL "https://<server_url>/arcgis/rest/services/<folder>/<service>/GPServer/webhook_handler/execute" failed. Exception: .

I do have an additional webhook configured on the same Hosted Feature Service and receive the payload successfully (and see the above logs, except for a successful execution). I have seen one reference that a GP Service could be incompatible with this type of workflow due to how ArcGIS Server ends up delivering it's payload, but I'm hoping that's not the case as my customer would like to keep everything within the GIS ecosystem if possible.

Re: Feature Service Webhook

KevinHibma — Mon, 21 Oct 2024 18:51:53 GMT

Hi @a-j-campbell

Two things happening here.

When using a synchronous GP Service, this execution type will not return a response until the task finishes. So if the service takes 20 seconds to finish, it wont return a message to the webhook process saying "200-success" until then. The webhook send process looks for that 200-success message, but in this scenario it's most likely timing. It only waits for so long. I suspect that's why you're getting that error.

Second, while you can send a message to a GP Service, unfortunately in ArcGIS Enterprise 11.1 there is an issue when using url-form-encoded that the payload is not actually sending as a proper encoded form. This means that the GP Service is going to have problems ingesting the incoming payload. It's possible you might be able to work around it, but the payload doesn't get encoded properly. So it'll become tougher to handle that.

Re: Feature Service Webhook

a-j-campbell — Mon, 21 Oct 2024 19:08:29 GMT

Hey @KevinHibma,

Thanks much for the prompt response!

In response to the first note - that makes sense. I'm not tied to synchronous processing for this service in particular, so can definitely explore moving it over to "Asynchronous" (assuming that might resolve this initial hurdle).

As far as the payload type is concerned, I am happy to handle any formatting issues via my own code at this time, although this is good to understand moving forward. Do you know, is url-form-encoded the only supported delivery method for GP services? This would make sense as I've been able to test my service using Postman with a successful execution run (assuming Postman is delivering the data in the correct encoding.

Thanks again!

Alex

Re: Feature Service Webhook

KevinHibma — Mon, 21 Oct 2024 19:16:18 GMT

Yup, Async should get you past that.

Postman will absolutely be doing the correct thing sending to your service.

You can set the webhook to send as JSON, but I'm 99% sure that you can't "get" the value into the gp service as JSON via this trigger. It needs to come in as a form which would match a parameter on the GP Service.

You can test sending your webhook payloads to https://webhook.site (free interest testing resource). This should give you an idea of how the payload as a form comes in. From there you might be able to determine how your code could handle that.

Re: Feature Service Webhook

a-j-campbell — Mon, 21 Oct 2024 19:26:27 GMT

Interesting.. that's all good to know as I had been wrestling with the appropriate delivery method (Webhook format vs GP parameter type(s)).

My additional webhook configuration on the same Service is actually through webhook.site, so glad I'm using something familiar.

Last question for the time being.. (thank you again). Switching my GP Service over to Async is now preventing me from updating the webhook registration via "/arcgis/admin/services/...".

Unable to create or update webhook. URL is not reachable: https://server_url/arcgis/rest/services/folder/WebhookHandler/GPServer/webhook_handler/submitJob for webhook GP Server

Could you elaborate on what Server is actually doing when you click "Create" or "Edit"? I know during initial webhook creation it will fire off a simple payload ({webhook: webhook name}) and I'm assuming it's looking for a 200 response to this initial request, and perhaps testing TLS authenticity? I am surprised to see that I'm unable to set my HookUrl to the same GP service, just the "/submitJob" endpoint for Async versus "/execute".

Re: Feature Service Webhook

KevinHibma — Mon, 21 Oct 2024 19:40:18 GMT

That's odd.

When you click create or edit, it's sending a HTTP message to that URL to check that 1) it responds and 2) its trusted (https /w valid security cert). If those checks fail, it wont allow you to create the webhook.

11.1 was our initial webhook release and we've fixed a few bugs since then. I'm not aware of any specific bugs in this area that might be causing this. I've created and seen others sending webhook payloads to GP Services; specifically async.

Re: Feature Service Webhook

a-j-campbell — Mon, 21 Oct 2024 19:47:15 GMT

Fascinating... thank you! I'll keep testing and see if I can pinpoint the issue to this environment. If so, we may push this until we can get on a newer LTS release (11.3, likely).

Glad to know it's possible! Thanks @KevinHibma

Re: Feature Service Webhook

Thomas_Fuhrmann — Tue, 22 Oct 2024 06:56:42 GMT

Update: We upgraded our ArcGIS Enterprise installation to 11.3, RabbitMQ has started correctly and now the feature service web hooks work!

Re: Feature Service Webhook

ColeHowell — Tue, 22 Oct 2024 12:34:08 GMT

Hello Kevin,

I've been having very similar issues in my ArcGIS Enterprise 11.2 environment. I have found that the config part of our webhook processors is not populated so I had my IT department do the restart as you laid out here but the config is still not populated. What should we try from here?

Thanks,

Re: Feature Service Webhook

a-j-campbell — Tue, 22 Oct 2024 14:23:18 GMT

Hey @ColeHowell! I wonder if you're hitting this BUG that I found during my investigation: https://support.esri.com/en-us/bug/webhook-processors-fail-after-upgrading-the-arcgis-data-bug-000164209. I can't speak to restart order, especially as I imagine this config is driven partially by Data Store, but it unfortunately could mean an upgrade to 11.3 is the best path forward. Just my 2 cents!