ArcGIS Server 11.0 Directory Traversal Vulnerability Patch

[DavidColey]

Hello - I just applied the ArcGIS Server 11.0 Directory Traversal Vulnerability Patch

https://support.esri.com/en/Products/Enterprise/arcgis-server/ArcGIS-Server/11#downloads?id=8063

On my (2 machine Windows Server 2016) 11.0 hosting site servers.

And now I am seeing hundreds and hundreds of SEVERE and WARNING level log errors pertaining to webhooks, yet I have no webhooks set up yet against any of the hosted feature layers:

WARNING Webhook log: FS Webhook processor init failed Connecting to queue : FS_Raw_Events_Queue failed.

SEVERE Webhook log: init WebhookProcessors failed. FS Webhook processor init failed.

WARNING Webhook log: Error in initializing webhook processor. init WebhookProcessors failed.

Has anyone else seen this?

@JonathanQuinn 

@JonEmch 

[DavidColey]

UPDATE

In our environment, the federated site registered with our enterprise is still at release 10.9.1.  I did apply the  

ArcGIS Server Security 2022 Update 2 Patch

https://support.esri.com/en/Products/Enterprise/arcgis-server/ArcGIS-Server/10-9-1#downloads?id=8064

which does contain a fix for the same issue - Directory traversal vulnerability in ArcGIS Server - 

This patch application is not generating any of the 3 unique log entries noted above.

[KevinHibma]

Hi @DavidColey 

I don't have much knowledge of that particular patch, but I'm doubtful that it caused the problem.

The message is indicating that GIS Server is failing to connect to the required components for webhooks. Are you able to restart the GIS Server? A restart will sometimes fix this problem.

Is there any chance the problem was there before you applied the patch and you didn't notice the logs? If so, we'd need to figure out what might be in the way of the connection. (closed ports perhaps)

[DavidColey]

Hello @KevinHibma - thanks for the reply.  Yes, a restart is something I would do after hours and will see if that helps. 

I use ArcGIS Monitor, so it is not possible that I would not notice the log:

ID

Category

Last Alert

Collection

Level

Status

H:M

Count

Groups

Counter Name

Rule

Counter Instance

Name

Comments

Counter Type

Int(min)

1

ArcGIS

10/07/2022 11:41 AM

Production

Warning

Open

17:44

72

1

Log-WARNING

> 0

Summary

ArcGIS Host

ArcGIS Errors

ArcGIS

15

2

ArcGIS

10/07/2022 11:41 AM

Production

Warning

Open

17:44

72

1

Log-SEVERE

> 0

Summary

ArcGIS Host

ArcGIS Errors

ArcGIS

15

As you can see, these 2 logs have been open for 17 hours and the error keeps rolling along.  So far, there is no impact to performance (cpu or memory) but it is hassle to find my real Severe and Warning level errors contained within.

[KevinHibma]

Thanks. That's sort of good news / bad news. I'm still skeptical the patch has caused the issue, but if you've only had the errors since applying the patch, it's now a question of how the communication was interrupted. I'll talk to a couple of colleagues and see if it's reproducible. 

Unfortunately, you'll see the messages once a minute as Server attempts to re-establish the connection. Please respond back with the result of restarting Server.

[DavidColey]

Sure - but: "I'm still skeptical the patch has caused the issue".  Well, the error log entries began moments after the patches were applied to the 2-machine host site cluster.  So if not the patch, then what? 

No one else could have attempted to create any type of the new (beta) webhook capabilities in portal at release 11, and I have not.  

If a restart does not address the log error entries, I may uninstall the patch updates and see if the errors continue.

[JonEmch]

Hey there David,

   Thank you for tagging me in this. Let me do some research on this and I will follow up.

[DavidColey · Accepted solution]

Hello @KevinHibma  - I was able to resolve this:

Restart arcgis server exe processes - no effect

Restart both windows servers - no effect

Uninstall patch - no effect

Reinstall patch - no effect

Then I decided to check the configuration of the webhook processes json in the 

....\config-store\system\webhookprocessors-config directory where I saw that this file uses a url connection to the datastore

"jdbcUrl":"jdbc:postgresql://DATASTOREMACHINE.BCC.SCGOV.LOCAL:9876/webhooks"

I checked that my port 9876 was open and unrestricted vis system resources and it is.

Regardless, I went ahead and with a restart of the:

ArcGIS Data Store service - log entries stopped.  No more errors. 

So basically a restart of the datastore service resolved some type of communication issue that may or may not have been coincidental with an ArcGIS Server service restart for any reason, not just application of this patch.

So I consider this resolved on my end at least for now.

@JonEmch 

[MDekkers]

We have recently seen this error message in some environments that had been upgraded to ArcGIS Enterprise 11. After some research it appeared that the firewall rules on the ArcGIS DataStore machine hadn't been update with the newly required ports (25672, 44369, 45671) for the added webhook functionality in ArcGIS Enterprise 11.
See these links for more information:
https://enterprise.arcgis.com/en/data-store/latest/install/windows/ports-used-by-arcgis-data-store.h...
https://enterprise.arcgis.com/en/portal/latest/administer/windows/about-arcgis-webhooks.htm 

[Thomas_Z2]

We experienced the same issue after we changed the PSA (primary site administrator) password.

Re-registering the ArcGIS Data Stores solved the problem for us.

It may be somehow related with the warning "Failed to log in. Invalid username 'siteadmin' or password specified." in our log files (or it solved by chance solved both issues). We were following this tutorial from Esri's technical support here.