ArcGIS Enterprise Log4j Vulnerability (CVE-2021-44228) Patch or Mitigation?

[Carl_Flint]

Good afternoon, is there any patches in the works or potential mitigation steps for the latest java log4j vulnerability (CVE-2021-44228)?  I know that GeoEvent server uses log4j and can assume some other enterprise server's or portal potentially do as well.  Any help would be appreciated in resolving this zero-day.

Thanks,

[MattFancher1]

Thank you @RandallWilliams. I see where I missed that in the instructions. Thank you for the quick response.

[Marion_CountyGIS_Coordinator]

The below article helps with data store :  https://support.esri.com/en/Technical-Article/000026949

[DavidColey]

More questions:

Can anyone to date verify that their system (on-premises base or distributed deployment) has in fact been exploited by this? 

How would we know, what would an exploit look like?

Finally, what is the specific version of python 3.5.x that should be installed on datastore?

[ErikLash1]

Not aware of any successful exploit of an ESRI system.

However, IoCs will rarely be visible if a perpetrator is successful.  Log4J vulnerability is much like a door - an access route vs the actual objective which is often to move lateral and establish a further exploit somewhere else in the system.

We analyze our IIS logs using an increasingly complex REGEX filtering for 200 returns. A full RCE chain will require round trip communication - each day our logs see thousands of attempts being made to leverage Log4J holes.

Every 200 return delivers a payload back to the offending IP and it is through following the successful 200 connections that also meet REGEX on which we undertake our exploit hunting exercises. We also combine that with system scanning using professional services that employ actual PoC to scan systems.

Entry prevention is really about layers. Network monitoring software, Perimeter firewalls, WAF, Load Balancer and Gateway configuration for security, system configuration to allow only the necessary ports and types of traffic and maybe most importantly component and version upgrades to the newest available versions when those versions are released.

Can not stress enough that users should follow ESRI guidance and upgrade to 10.9.1 to take advantage of better security under the hood within the actual components included in the 10.9.1 distribution. That's the last line of defense should someone make it through the other security layers that organizations implement.

We installed/uninstalled Pro on our datastore machine to undertake the mitigation scripting. Not the most elegant Python solution but easy to accomplish.

[rshihab]

Hi David

Download python v3 from the link https://www.python.org/ftp/python/3.10.1/python-3.10.1-embed-amd64.zip

to the datastore machine and run the command as per Esri support site

thanks

Ramla

[ErikLash1]

To provide an actionable answer to the question of "what does an exploit look like" there are a few quick and easy things that GIS administrators can check in their systems:

• Most adversaries use a different IP to scan than they do to listen for incoming victims machines - tracking listening IPs from known treat actors is far more valuable to than scanning IPs.
• An outbound connection or listening wait from a GIS machine to a known threat actor listening IP may be the easiest IoC to scan for. This can easily be accomplished using NetStat in windows.
• Activity or connection from a GIS machines Java processes that does not conform to regular ESRI implementation parameters such as Java child processes calling wget, curl, or powershell commands. This is another easy IoC review that can be undertaken by GIS administrators.
• Outbound LDAP and RMI and rogue JRMI or LDAP requests to external servers is a little more complex review but can be monitored in real time. If possible after determination if LDAP connections are required on your machines, blocking those connections is a good way to close the door entirely on that vector.
• Scanning for webshells is super important. Webshells are not only an IoC for successful Log4J exploitation they are extremely common in other attacks.
• Scan for and detect payload execution on a machine for post-exploitation activities. Many of these activities will be easily noticed even if endpoint security is not up to date - crypto-currency mining software (one of the most common early post exploit activities) will consume the systems resources. Others that will generate unique signatures include Mirai and BazarLoader.

Let's keep the list going - what else will a successful exploit look like and what are the methods you are undertaking as you mitigate on your systems?

[DavidColey]

Thank you for this @ErikLash1 .  That is a lot to take in and take under advisement and will be super helpful to pass on to my security team - IIS monitoring on the server hosting web adaptor will be in their realm.  Our 10.9.1 distributed deployment resides behind an F5, and they tend to be very aggressive in terms of port and log scans as it is.  I myself actively monitor performance and error logs through Monitor for the entire deployment and have yet to see anything like a resource hit or unusual errors or calls like you describe.

[RandallWilliams]

HI All,

Sorry for the lack or recent participation. As you can imagine, we've been heads down with Log4J and other tactical and strategic issues software security issues here at Esri. 

To answer some questions in no particular order:

@ErikLash1 @DavidColey 

Esri has not yet heard from a customer who has been exploited via log4j of any version in any of our software. 

We (Software Security and Privacy) AND the core ArcGIS Enterprise development team used white box testing techniques (walking the source code) while testing these 5 recently logged CVEs and we have not been able to exploit these either. We do not believe that ANY of these log4j issues are exploitable in our software (mitigation by design), but we are nonetheless producing formal patches for all versions in current support status since December 09 2021.

In terms of what an exploit would look like:

An exploit would most likely look like one of the bundled JREs that ship with ArcGIS Enterprise invoking some windows executable - like Javaw.exe --cmd /c c:\path\to\some\exe --some --argument. 

Note that ArcGIS Enterprise (as designed) itself will call some exes by default - like we call (for example) pg_isready to check the health of the internal database. 

If you see a JAVAW instance calling wget or ftp or similar and pulling down a file...that would concern me. 

[DavidColey]

Super, thank you for this @RandallWilliams .  This will also be super helpful to pass along to the security team. 

[Ikebana]

hello!

If you run the log4shellmitigation script on "arcgisportal-content" catalog the script tells you that it's need patching. Have anyone tried run the script on this catalogs? To be clear, it's not the installationfolders, just the content folders.