Notebook Server 11.5 Webhook Receivers

[Justin_Greco]

I noticed that in the release notes for 11.5 there is now the ability to set up a webhook receiver.  The documentation is very limited.  It gives you instructions for creating the receiver, which includes specifying the notebook you want the webhook to trigger, a secret (which I think can just be random 32 digit string), an API token (which I assumes means I need to generate a token on Portal and include that) and signature headers (documentation has "x-esrihook-signature,x-hub-signature-256" as an example.  This gives me a webhook URL.

I took that webhook URL and used it to set up a webhook on a feature layer (does have edit tracking enabled) and set it to FeatureCreated.  My expectation is that the notebook would run after I created a feature, which I don't have any evidence that it ran.

Since the documentation is limited, I am thinking something is not configured correctly either when creating the webhook receiver or with the URL I am setting on the feature service webhook.  Is anyone aware of any additional documentation or blog posts about this feature?

[KevinHibma]

Thanks, Justin, we'll use your question as a motivator to get more information out.

For your issue. A few things to try:

1. You can setup a second webhook with the exact same parameters as your Notebook Webhook Receiver, just point it some place else, like https://webhook.site/  - This should give an idea if it's working
2. Alternatively, ArcGIS Server debug logs should show if the webhook is being dispatched, or if there are errors
3. Finally, if you set debug logs in the Notebook Server Administrator, you should see logging related to the notebook triggering.

For the most part, setting up the receiver is the bulk of the work. The only thing you didn't note was the webhook secret key. Did you use the matching 32 character secret key while creating the webhook? This same value must be set when creating the webhook.

[BrianJohnston4]

I have set a webhook up and webhook receiver and made changes to the hosted feature layer attributes.  Each time I find no activity in the notebook server logs.  I also found a notificationStatus end point on the webhook URL as well, but nothing is shown there to indicate activity.  However, if I remove the secret value from webhook, I see this in the notebook server log: "The signature header value is invalid" after I update an attribute on the feature layer.  That proves the webhook is firing and attempting to call receiver, but apparently when successful it silently succeeds.  I can find no documentation explaining what the payload looks like, nor how to get to the payload in a notebook.  https://webhook.site/  is blocked so I am going to make a custom input in GeoEvent and see if I can figure out more.  Definately need more material about this stuff from Esri to actually get it to work.  It's been a real challenge since the documentation said make an "API key" to create the receiver, when in reality it should of said "generate a Portal token".  That one cost me hours to figure out. 

[Justin_Greco]

Are you sure about needing to generate a portal token instead of an API key?  I was at a session at the Developer Summit where they did a demo of creating a webhook receiver.  They definitely created an API key.  Which I have tried, but still can't get the webhook to trigger the receiver. 

I followed what was done in the demo and still can't get it to work.  Though the demo was using 12.0 where you do it all in the Portal UI, instead of in the admin API of Notebook Server like I have to do at 11.5.

[BrianJohnston4]

Justin,  I am not able to create a webhook receiver on a hosted feature layer with a "developer API key".  I have tried and found the key can login, access the notebook, but cannot update it.  Apparently it needs to be able to update it to work.  I even had Gemeni create a small script to demonstrate the issue (attached).  However, if I generate a Portal token, put it into the apiToken parameter, it I am able to create a receiver.  In addition, I tested the webhook and receiver and proved it does fire when I change data in the hosted feature layer.  This is how I tested the notebook: 

import os
home_dir = '/arcgis/home'
new_dirname = 'webhook_receiver_fired_1'
my_dir = os.path.join(home_dir, new_dirname)
os.mkdir(my_dir)

[KevinHibma · Accepted solution]

@BrianJohnston4 @Justin_Greco 

You absolutely need to be using a Portal API Key, and yes, with the inclusion of the UI parts in 12.0, the general workflow to set this up does get easier. For both 11.5 and 12.0, in the Portal, if you do New Content and select Developer Credentials then API Key this is your start point. What's important there is making sure the correct Notebook specific privileges are set, as well as the Notebook item (python code) and any other items that Python code will access.

reference:  https://enterprise.arcgis.com/en/notebook/11.5/administer/linux/automate-notebook-execution.htm#ESRI...

Before we got the UI rolling, I had a script I used to perform this NBS Webhook Receiver create workflow. If you have the Python API installed, you can give it a try. Look through and check for the hardcoded references and update as needed (URLs, credentials, itemIds, etc)

edit: As I re-look at the provided sample script, I'm giving privileges to premium privileges. I forget exactly, but this might be necessary at 11.5. I can't recall exactly why I have granted those. I do know at 12.0 and 12.1 (to be released in a few months), you don't need these.

If you have the receiver setup and are sending messages to it, but it's not executing as you expect, head to the Notebook Server /admin-jobs and search out the job. The output here is pretty good to show errors (permissions, code problems, etc).

https://notebookserver.example.com/gis/admin/system/jobs

To get/use the payload in the Notebook (Python code), you need to "use" the webhookPayload variable. The receiver getting the webhook payload auto-magically injects this into your code and you simply use it by reference. At 11.5/12.0 you need to cast it as JSON to use it (json.loads(webhookPayload). At 12.1 it comes in as an object and no need to cast.

I'm still catching up after Developer and Technology Summit, but I plan to add these details to https://github.com/Esri/webhooks-samples in the next few days.

[BrianJohnston4 · Accepted solution]

@Justin_Greco , @KevinHibma 

A colleague showed me the solution.  When creating the API Key you must set access to the notebook item with the "Restrict Item Access" button (see screenshot).  The "Edit Item Access" button in the settings does not work.  

[Justin_Greco]

Thanks Brian, I believe that is why it wasn't getting through

[Justin_Greco]

@KevinHibma Thanks for suggesting looking at the jobs, turned out it was executing the notebook, but I had an error in my notebook. 

One other question is should I be able to use the receiver URL with an ArcGIS Online feature layer?  I set up a webhook on an AGOL feature layer, specified the receiver URL, the secret and set the content type to application/json.  However, its not making it to the receiver.  

[KevinHibma]

Interesting. I want to say this should work, but if the ArcGIS Online feature layer webhook isn't sending the secret key, or the payload exactly the same as the ArcGIS Enterprise Hosted Feature layer webhook, then I'd accept that it might fail.  Good catch on switching to application/json from the x-form-encoded. I'm sure that's a necessary configuration step.

Let me talk with the team - I personally haven't tested this, but will check and see if anyone else has.