Feature Service Webhook

[Thomas_Fuhrmann]

Hi all,

I'm trying to set up a Feature Service Webhook in ArcGIS Enterprise 11.1. The webhook should send a POST request to a custom receiver when a feature is created. Setting up the webhook is possible, but when I create a feature there isn't any request sent.

I tested with the same custom receiver and an ArcGIS Online webhook. In this situation the post request is sent correctly. I assume that it can't be related to CORS since it works in this situation and I also added the hostname of my ArcGIS Enterprise server as an allowed origin to my custom receiver's web server (Apache). And I'm able to send a post request to the custom receiver from the ArcGIS GIS server machine manually.

Port 443 is open in the ArcGIS Enterprise network. The whole ArcGIS Enterpise installation is behind a KEMP reverse proxy/load balancer.

Does anyone have any ideas?

[KevinHibma]

I'd turn debug logs on in the server. Perform your feature service action (add feature), and then check the logs. You should see messages specific to webhooks. Based on your explanation it sounds like you will NOT see the webhook dispatched message. But you might see errors or other log messages that could be helpful.

[Thomas_Fuhrmann]

Thanks for the reply @KevinHibma! Unfortunately there aren't any debug logs related to this service after performing the feature service action (create feature).

I found the following documentation which could be relevant in this case:

• 25672 and 44369 (TCP)—These ports must be available (free) on the relational data store machines because service webhooks require them.
• 45671 and 45672 (TCP)—These ports on the relational data store machines must be available to allow communication with the ArcGIS GIS Server sites because service webhooks require them.

I checked these ports and they aren't open yet.

[Thomas_Fuhrmann]

@KevinHibma After checking our firewall settings, I tested feature service webhooks again. Unfortunately there is no post request sent after creating a new feature. I tested the same webhook receiver with organization webhooks and it works fine. The server logs do not show any error messages.

Do service webhooks require a running RabbitMQ service?

I checked under Windows services and could not find any running RabbitMQ processes.

Is there any configuration parameter in ArcGIS Server that allows me to enable webhooks?

[KevinHibma]

Hi Thomas,

You won't see a "RabbitMQ" process by name. It's wrapped in another process. But I wouldn't go real deep looking here.

You said you're using ArcGIS Enterprise 11.1? When you check your Server debug logs, do you see any errors related to webhooks? If not, thats good. If so, can you paste them here? If you're getting errors I'd suspect you would be seeing them quite often.

If no errors, within the Server administrator, can you check /server/admin/system/webhookprocessors  (you'll need to type the webhookprocessors part in, it's not a published API). If you check the config, is it populated? If not, can you restart you ArcGIS Data Store, and then your ArcGIS Server, then Portal  (the restart is just to get everything into a state where that config would have information. Note - if it is already filled in, just try restarting ArcGIS Server.

Further to your question, no, there is no configuration parameter you need to set or modify to enable webhooks. The config I mention above would automatically be filled in by the system: you cannot manually update it.

[Thomas_Fuhrmann]

Thanks for the informative and quick reply, @KevinHibma ! In the meantime we upgraded to version 11.3. I checked the config for the webhookprocessors and it's already populated. Our ArcGIS server automatically restarts once a week.

In ArcGISDatastore\etc\queuestore-config.json I found the following config. The datastore.release version should be 11.3, datastore.release.rabbitmq should be 3.13.0, datastore.release.rabbitmq.erlang should be 26.2 and datastore.lastknownstatus should be Started:

{
"datastore.release": "11.0.0",
"datastore.release.configstore": "1.5",
"datastore.release.rabbitmq": "3.9.13",
"datastore.release.rabbitmq.erlang": "24.2",
"datastore.name": "e3t13dt",
"dir.dbdata": "D:\\ArcGISDatastore\\rabbitmqdata",
"db.host": "GIS-DS01.GBA.GEOLBA.AC.AT",
"mode": "primaryStandby",
"replication.role": "PRIMARY",
"datastore.registered": true,
"localstore.lastknownstatus": "Upgrading",
"owningsys.type": "GISSERVER",
"healthcheck.enable": true,
"datastore.lastknownstatus": "Stopped"
}

I checked the logs of our Data Store machine and found the following entries related to webhooks:

<Msg time="2023-01-26T14:38:12,777" type="WARNING" code="110616" source="Data Store" process="8052" thread="17" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="" requestID="">The Webhooks database for the site "382a15d6-5c5e-44ba-89fc-c3bef95bd0d3" is not created and registered.</Msg>
<Msg time="2023-05-04T13:10:45,52" type="SEVERE" code="111902" source="Data Store" process="5300" thread="27" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="" requestID="">Error configuring the queue store. null</Msg>
<Msg time="2023-12-21T13:55:31,925" type="SEVERE" code="111902" source="Data Store" process="5896" thread="24" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="" requestID="">Error configuring the queue store. C:\ArcGIS\DataStore\framework\runtime\rabbitmq\erl-24.2\bin\erl.ini (The system cannot find the specified path)</Msg>
<Msg time="2024-06-12T14:04:50,77" type="SEVERE" code="111902" source="Data Store" process="8176" thread="18" methodName="" machine="GIS-DS01.GBA.GEOLBA.AC.AT" user="" elapsed="0.0" requestID="">Error configuring the queue store. Cannot invoke "com.esri.arcgis.datastore.client.UpgradePrerequisiteCheckResult$StoreCheckResult.getUpgradable()" because "&lt;local11&gt;" is null</Msg>