Feature Service Webhook

[Thomas_Fuhrmann]

Hi all,

I'm trying to set up a Feature Service Webhook in ArcGIS Enterprise 11.1. The webhook should send a POST request to a custom receiver when a feature is created. Setting up the webhook is possible, but when I create a feature there isn't any request sent.

I tested with the same custom receiver and an ArcGIS Online webhook. In this situation the post request is sent correctly. I assume that it can't be related to CORS since it works in this situation and I also added the hostname of my ArcGIS Enterprise server as an allowed origin to my custom receiver's web server (Apache). And I'm able to send a post request to the custom receiver from the ArcGIS GIS server machine manually.

Port 443 is open in the ArcGIS Enterprise network. The whole ArcGIS Enterpise installation is behind a KEMP reverse proxy/load balancer.

Does anyone have any ideas?

[KevinHibma]

I'd turn debug logs on in the server. Perform your feature service action (add feature), and then check the logs. You should see messages specific to webhooks. Based on your explanation it sounds like you will NOT see the webhook dispatched message. But you might see errors or other log messages that could be helpful.

[Thomas_Fuhrmann]

Thanks for the reply @KevinHibma! Unfortunately there aren't any debug logs related to this service after performing the feature service action (create feature).

I found the following documentation which could be relevant in this case:

• 25672 and 44369 (TCP)—These ports must be available (free) on the relational data store machines because service webhooks require them.
• 45671 and 45672 (TCP)—These ports on the relational data store machines must be available to allow communication with the ArcGIS GIS Server sites because service webhooks require them.

I checked these ports and they aren't open yet.

[Thomas_Fuhrmann]

@KevinHibma After checking our firewall settings, I tested feature service webhooks again. Unfortunately there is no post request sent after creating a new feature. I tested the same webhook receiver with organization webhooks and it works fine. The server logs do not show any error messages.

Do service webhooks require a running RabbitMQ service?

I checked under Windows services and could not find any running RabbitMQ processes.

Is there any configuration parameter in ArcGIS Server that allows me to enable webhooks?

[KevinHibma]

Hi Thomas,

You won't see a "RabbitMQ" process by name. It's wrapped in another process. But I wouldn't go real deep looking here.

You said you're using ArcGIS Enterprise 11.1? When you check your Server debug logs, do you see any errors related to webhooks? If not, thats good. If so, can you paste them here? If you're getting errors I'd suspect you would be seeing them quite often.

If no errors, within the Server administrator, can you check /server/admin/system/webhookprocessors  (you'll need to type the webhookprocessors part in, it's not a published API). If you check the config, is it populated? If not, can you restart you ArcGIS Data Store, and then your ArcGIS Server, then Portal  (the restart is just to get everything into a state where that config would have information. Note - if it is already filled in, just try restarting ArcGIS Server.

Further to your question, no, there is no configuration parameter you need to set or modify to enable webhooks. The config I mention above would automatically be filled in by the system: you cannot manually update it.

[Thomas_Fuhrmann]

Thanks for the informative and quick reply, @KevinHibma ! In the meantime we upgraded to version 11.3. I checked the config for the webhookprocessors and it's already populated. Our ArcGIS server automatically restarts once a week.

In ArcGISDatastore\etc\queuestore-config.json I found the following config. The datastore.release version should be 11.3, datastore.release.rabbitmq should be 3.13.0, datastore.release.rabbitmq.erlang should be 26.2 and datastore.lastknownstatus should be Started:

{
"datastore.release": "11.0.0",
"datastore.release.configstore": "1.5",
"datastore.release.rabbitmq": "3.9.13",
"datastore.release.rabbitmq.erlang": "24.2",
"datastore.name": "e3t13dt",
"dir.dbdata": "D:\\ArcGISDatastore\\rabbitmqdata",
"db.host": "GIS-DS01.GBA.GEOLBA.AC.AT",
"mode": "primaryStandby",
"replication.role": "PRIMARY",
"datastore.registered": true,
"localstore.lastknownstatus": "Upgrading",
"owningsys.type": "GISSERVER",
"healthcheck.enable": true,
"datastore.lastknownstatus": "Stopped"
}

I checked the logs of our Data Store machine and found the following entries related to webhooks:

<Msg time="2023-01-26T14:38:12,777" type="WARNING" code="110616" source="Data Store" process="8052" thread="17" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="" requestID="">The Webhooks database for the site "382a15d6-5c5e-44ba-89fc-c3bef95bd0d3" is not created and registered.</Msg>
<Msg time="2023-05-04T13:10:45,52" type="SEVERE" code="111902" source="Data Store" process="5300" thread="27" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="" requestID="">Error configuring the queue store. null</Msg>
<Msg time="2023-12-21T13:55:31,925" type="SEVERE" code="111902" source="Data Store" process="5896" thread="24" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="" requestID="">Error configuring the queue store. C:\ArcGIS\DataStore\framework\runtime\rabbitmq\erl-24.2\bin\erl.ini (The system cannot find the specified path)</Msg>
<Msg time="2024-06-12T14:04:50,77" type="SEVERE" code="111902" source="Data Store" process="8176" thread="18" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="0.0" requestID="">Error configuring the queue store. Cannot invoke "com.esri.arcgis.datastore.client.UpgradePrerequisiteCheckResult$StoreCheckResult.getUpgradable()" because "&lt;local11&gt;" is null</Msg>

[ColeHowell]

Hello Kevin,

I've been having very similar issues in my ArcGIS Enterprise 11.2 environment. I have found that the config part of our webhook processors is not populated so I had my IT department do the restart as you laid out here but the config is still not populated. What should we try from here?

Thanks,

[a-j-campbell]

Hey @ColeHowell! I wonder if you're hitting this BUG that I found during my investigation: https://support.esri.com/en-us/bug/webhook-processors-fail-after-upgrading-the-arcgis-data-bug-00016.... I can't speak to restart order, especially as I imagine this config is driven partially by Data Store, but it unfortunately could mean an upgrade to 11.3 is the best path forward. Just my 2 cents! 

[a-j-campbell]

Hey @KevinHibma! I'm going to revive this thread in hopes you'll be able to help me out! I've been working on developing a custom webhook receiver using a synchronous GP service and believe I've got things correctly written to handle a Hosted Feature Service webhook (ArcGIS Enterprise 11.1) but am getting a generic error in the debug logs when I try to trigger the webhook.For some additional context, these are my Webhook Settings:

HookUrl	https://<server_url>/arcgis/rest/services/<folder>/<service>/GPServer/webhook_handler/execute
Active	true
Change Types	FeaturesUpdated
Payload Format	json
Content Type	application/x-www-form-urlencoded
Signature Key
Schedule Info	{"name":"","state":"enabled","startAt":1729524167012,"recurrenceInfo":{"interval":20,"frequency":"second"}}
Creation Time	1729524191497
Last Update Time	1729528085234

Based on this discussion, I'm seeing the "calling" message for this webhook:

Webhook log: Calling webhook url ==> https://<server_url>/arcgis/rest/services/<folder>/<service>/GPServer/webhook_handler/execute for GP Server.

But don't see the "dispatching" message. I do eventually get the following (without an Exception trace):

Webhook log: Calling webhook URL "https://<server_url>/arcgis/rest/services/<folder>/<service>/GPServer/webhook_handler/execute" failed. Exception: .

I do have an additional webhook configured on the same Hosted Feature Service and receive the payload successfully (and see the above logs, except for a successful execution). I have seen one reference that a GP Service could be incompatible with this type of workflow due to how ArcGIS Server ends up delivering it's payload, but I'm hoping that's not the case as my customer would like to keep everything within the GIS ecosystem if possible.

[KevinHibma]

Hi @a-j-campbell 

Two things happening here.

When using a synchronous GP Service, this execution type will not return a response until the task finishes. So if the service takes 20 seconds to finish, it wont return a message to the webhook process saying "200-success" until then. The webhook send process looks for that 200-success message, but in this scenario it's most likely timing. It only waits for so long. I suspect that's why you're getting that error.

Second, while you can send a message to a GP Service,  unfortunately in ArcGIS Enterprise 11.1 there is an issue when using url-form-encoded that the payload is not actually sending as a proper encoded form. This means that the GP Service is going to have problems ingesting the incoming payload. It's possible you might be able to work around it, but the payload doesn't get encoded properly. So it'll become tougher to handle that.