ArcGIS and CVE-2021-44228 Apache Log4j 2

[RandallWilliams]

Regarding the recently announced 0 day CVE-2021-44228 (aka Log4Shell aka LogJam):

Details regarding a new security vulnerability identified as CVE-2021-44228 (aka Log4Shell aka LogJam) were released on December 30.

This issue is generating considerable media attention, and is currently Esri Product Security Incident Response Team's highest priority. 

Please follow the advisory posted on the ArcGIS Trust Center for the latest regarding this issue. 

[MichaelVolz]

Will patches be made available for ArcGIS 10.7.1 and earlier?  The article does not answer that question.

[DavidHoy]

For those in Non-US time zones

The bulletin ArcGIS and CVE-2021-44228 Apache Log4j 2 (esri.com)

was updated by @RandallWilliams at 6:30 pm PT

WAF rules for Azure & AWS and how to apply them now listed in TRUST doc

ArcGIS Enterprise Web Application Filter Rules

[RobertDarlow]

Given the statement: 

"ArcGIS Enterprise versions 10.7.1 and earlier are potentially vulnerable (independent of using an Apache web server), however there is no exploit code available for ANY version or the components of a base ArcGIS Enterprise deployment..."

To what extent does that help, and how? Similarly with the other mitigations that don't go as far as actual upgrades (10.8 or 10.9). i.e. WAF and no-lookup.

[ccharping_patrickco]

@RandallWilliams what's the timeline for the patch releases? I have clients I work with who my team is already beginning to plan emergency upgrades for this week due to the news. Ideally, it would be better if we could simply patch and deal with upgrades on a normal schedule. Is there any hope that patches will become available this week perhaps? Especially for 10.7.1 environments.

[RandallWilliams]

All,

We get that this is a frustrating time. Large scale vulnerabilities like this are difficult for customers and vendors alike. We know that customers can't usually upgrade at the drop of a hat. Considerable developments have occurred over the night yesterday and continue to happen today. We've received a lot of feedback (A LOT OF FEEDBACK) and are working to provide updates as soon as we can.

[JohnHu]

I just finished our 10.9.0 Enterprise ArcGIS base deployment in our local network and on internal window server in November 2021.  Is this good enough or do I have to upgrade to 10.9.1 ?  Thanks.

[SimonSchütte_ct]

"Several ArcGIS Enterprise components contain the vulnerable log4j library, however there is no known exploit available for any version of a base ArcGIS Enterprise deployment (including the ArcGIS Server, Portal for ArcGIS, and ArcGIS Data Store components) or stand-alone ArcGIS Server at this time."

https://www.esri.com/arcgis-blog/products/arcgis-enterprise/administration/arcgis-software-and-cve-2...
The blog has been updated a few minutes ago.
The latest blog posts provides Log4Shell mitigation scripts that are strongly recommended for all installations of ArcGIS Enterprise and ArcGIS Server.

[Anonymous User]

It appears as though this vulnerability affects Log4j beginning at version 2.0-beta9. ArcGIS Server 10.4.1 appears to use earlier versions that are not impacted by this vulnerability. Can Esri confirm that this is the case?

[RandallWilliams]

@Anonymous User 

While we didn't test versions lower than 10.6.1 because they are so out of date, the script provided should work and you should use it. 

Then you should upgrade and install all the security patches you've missed out on over the last few years.

[Anonymous User]

Hi @RandallWilliams,

The script did work (once I installed an appropriate version of Python 3), but no files were identified because none of the installed log4j files match the patterns that the script is searching for.

I believe what this means is that the version of log4j used in ArcGIS Server 10.4.1 is not affected by this vulnerability, according to the Apache documentation.

[ccharping_patrickco]

My team just tested the script patches against 10.7, 10.7.1, and 10.8 Enterprise versions and the script works exactly as described, so long as the instructions are followed. Thank you @RandallWilliams and ESRI for providing this hotfix with urgency!

You guys rock!

[MichaelVolz]

Does the script leave the .jar files behind after the script is run, as I do see the same file with a .bak suffix?

[JohnHu]

When I run the first ArcGIS server python script,  I did see the list of JAR files but I encountered permission issue in the "--delete" part.  I need to resolve the permission issue and try again.

[JohnHu]

Hi ccharping_patrickc

Any problem when you run "--delete" part ? 

[ccharping_patrickco]

@JohnHu The following steps should eliminate that traceback error. First stop the ArcGIS Service that you are attempting to patch (aka portal, server, or datastore), next run python 3 as admin. As a test, our team attempted to run the script without stopping the services on a dev environment just to see what would happen (against the instructions). Doing so gave us the same permission issue you are seeing. I don't believe that permission lock will release or reflect back accurately unless you also close out of python and come back in fresh after stopping said service(s). Give that a shot.

[MichaelVolz]

I was using a domain account to log into the server as an admin, but I still did not have all the permissions needed.  I needed to grant full control to the user for the folders where the files would be impacted in order for the delete script to run successfully.

[ZianChoy]

The blog post says "base ArcGIS Enterprise deployment (including the ArcGIS Server, Portal for ArcGIS, and ArcGIS Data Store components) " and then "Base ArcGIS Enterprise components ... are therefore not vulnerable to CVE-2021-4104."

Does the phrase "Base ArcGIS Enterprise components" refer to the list of "base ArcGIS Enterprise deployment" bits? Does the phrase "Base ArcGIS Enterprise components" encompass any additional components?

[SpatialAmolOne92]

We have successfully run the script on our 10.8, 10.8.1, and 10.9 BEDs. Thank you @RandallWilliams and ESRI!

Putting it out here if anyone else encounters a similar behavior - We encountered an error "that another process is using the log4j-core.2.11.jar" file when we ran the -delete command even after making sure that ArcGIS Server service and GeoEvent service were stopped. We noticed that Zookepper's JDK process was still running which was blocking the script. Once we manually killed the process, we were able to successfully run the script. 

Best!

[GuglielmiNicole]

@RandallWilliams @Anonymous User  - we too had the same experience when applying the script to our legacy 10.3.1 environment.   Does this mean the vulnerability doesn't exist at 10.3.1?  Our security resources are asking for me to confirm whether or not there is anything further we need to do for this environment.  Any assistance you can provide would be great!  Thanks

[AYUSHYADAV]

Hi @RandallWilliams ,

Do we need to take any actions on Notebook Server version 10.8.1 as well? If so, will the ArcGIS Server scripts work? 

[SimonSchütte_ct]

@GuglielmiNicole
Read https://community.esri.com/t5/arcgis-enterprise-questions/arcgis-enterprise-log4j-vulnerability-cve-...

and surrounding comments on this topic.

[ccharping_patrickco]

@GuglielmiNicole I likewise was running the scripts on numerous production servers last night, two of which were older 10.3.1 versions. I also did not find any files to be patched, my assessment would be the same as yours that the vulnerability does not exist at the 10.3.1 version of AGS.

[SaraSiskavich]

I blasted forward from 10.4.1 to 10.8.1 last night and ran the mitigation scripts with no problem.  I'm looking for guidance on implementation WAF on an Amazon installation with a classic load balancer.  TIA.

[RandallWilliams]

@SaraSiskavich There is a WAF guide on https://trust.arcgis.com. It's been updated in response to this issue. It's under the documents tab - in the customer-only docs tile. You need to sign in w/ your ArcGIS account to access.

[SaraSiskavich]

@RandallWilliams thanks I did read through that.  It references the AWS guidance on WAF, which by my quick read can be applied to application load balancers and not classic load balancers.  Our classic load balancer dates back to when we implemented ArcGIS for Server Workgroup 10.4.1 using the ESRI AMI from AWS Marketplace.  

Further googling around hints that you can tie your classic LB to Cloudfront and apply WAF that way, but I'm out of my league at this point.  Is this the right thing to do?

[BenjaminBlackshear]

Our IT is asking whether ESRI has run the Logpresso CVE-2021-44228 scanner on ArcGIS Enterprise products.

or how it was determined that there is no known exploit available for any version of a base ArcGIS Enterprise deployment.

Can anyone from ESRI comment on this?

[RobertDarlow]

Previously, the Trust Centre guidance recommended upgrading to 10.8 upwards and that a patch/script would become available for earlier versions. I took from that that 10.8+ were deemed to be invulnerable to the Log4j issue. 

Now, the guidance makes no mention of Ent/Server versions and wholly focusses on running the script. Can anyone confirm whether there are still vulnerabilities with versions 10.8 and above? i.e. without also running the mitigation scripts.

@RandallWilliams 

[MichaelVolz]

Robert:

What version of enterprise are you currently using?

Is that version installed with log4j v 2.15, as v2.15 as per industry is not deemed to be secure and you need to upgrade to at least v2.16 (I've read that v2.17 is now available) unless you run the python scripts?  Hopefully Randall can provide additional clarification.

[RobertDarlow]

@MichaelVolz All the important stuff was upgraded to the latest version.

[RandallWilliams]

@RobertDarlow I can provide clarity. 

On Friday, Dec 10 (when this issue broke) the guidance provided by Apache indicated that apps that use Java 8.0.121+ were protected by built in protections in java that prevented the deserialization and execution of remote code. They also indicated that an environment variable would mitigate. Researchers quickly discredited those mitigation measures, so we removed them from our guidance. As you can tell, this is still an evolving situation - Apache released Log4J 2.17 Friday, December 17. 

Run the scripts and check the advisory. 

[Anonymous User]

@RandallWilliamsWhen did Esri's guidance change on this and how can we be notified of changes to the guidance? Had I not come to this thread to check, how would I have discovered that the guidance changed?

[RandallWilliams]

@Anonymous User It changed December 15. Subscribe to the RSS feed on the ArcGIS Trust Center.  

[BrandonWebb]

@RandallWilliams We have run the script against all installations of ArcGIS Enterprise 10.8.1 successfully for Server, Portal, and Datastore.  However, our security scans are still showing vulnerabilities of Apache Log4j 1.2 JMSAppender Remote Code Execution (CVE-2021-4104) on ElasticSearch, Spark, and Zookeeper embedded into the installation folders.

C:\Program Files\ArcGIS\DataStore\framework\runtime\elasticsearch_2.3.2\lib\log4j-1.2.17.jar

C:\Program Files\ArcGIS\Server\framework\runtime\zookeeper\lib\log4j-1.2.17.jar

C:\Program Files\ArcGIS\Server\framework\runtime\spark\jars\log4j-1.2.17.jar

Does  ESRI have a recommendation for these?  To my knowledge, these are included in the base installation of these applications, however the statement on the blog post says the base application is not vulnerable to CVE-2021-4104.

Base ArcGIS Enterprise components do not utilize and are therefore not vulnerable to:
–  Log4j 1.2 JMSAppender – CVE-2021-4104

Thanks!