A digital 'vaccine' for Log4Shell [ML BSide]

Transcription edited by @hakinadey

[Ran] Hi and welcome to CyberReason’s Malicious Life B-Sides, I’m Ran Levy.
You know, when we started Malicious Life some 4 years ago, I was wondering how much stuff is there to talk about in a podcast about cyber security? I mean, I knew there were some great stories to be told. I wrote a book about the history of cyber security back in 2012, so I did quite a lot of research back then. But how many great stories are there? Enough for 50 episodes? 100? 150? Well, here we are, 4 years later, in episode number 145, I think, and we’re not even close to running out of ideas.
But if there ever was a chance that we’ll run out of stories in the foreseeable future, along came Log4Shell and made sure we won’t. Why? Because, as Amit Yoran, CEO of Tenable, said, Log4Shell is by far the single biggest, most critical vulnerability ever. It seems that there are so many applications and so many organizations affected by Log4Shell that we’re probably going to hear that name being thrown around quite a bit in the next few years.
So, in today’s B-side, Log4Shell. What it is, and what can you do about it? Nate Nelson, our senior producer, talks to Yonatan Striem-Amit, CTO and Co-Founder of Cybereason, our sponsor, about the vulnerability and about the interesting quote-unquote vaccine that Cybereason released for it, a rather unusual piece of software that interestingly uses the same vulnerability that the bad guys are exploiting for their nefarious purposes to close the breach and make your server safe again, at least temporarily, with no downtime whatsoever.
Enjoy the interview.

[Nate] First, let’s just lay down some context. What is Apache Log4j, and how common is it?

[Yonatan] The Apache Log4j library is one of the most ubiquitous libraries in the Java world. It’s basically the go-to solution for most Java developers when they want to look for logging facilities. Now, logging being the most basic way for application debugging and auditing, so it’s been used in almost every Java server and Java application out there.
So, in that sense, if the conversation today is about three to four billion devices running Java. I would say the majority of them, to some degree, use Log4j within one of their components for a logging component.

[Nate] So, how could an attacker exploit Log4j?

[Yonatan] The developers for Log4j thought many years ago, when they just started the first version of the Log4j2 library that I think exactly what year it came out, many, many years ago, they thought that it would be a good idea to allow people who write logs to also add customized formatting for messages, seemingly a good idea at the time. What they didn’t realize is that it is very very common for developers to take input that they receive from the internet and write them in their logs.
I’ll give you the most canonical example. If you have a server that asks for a username and password in order to begin operation, it is considered best practice that on every login attempt, whether successful or benign, you would write to the log saying the user, and that would then use the username, has attempted to log in to your system and either succeed or fail. However, by this point, hackers are able to say, let us use as a username, as this example, one of the ways that I had load custom formatting code, and that will trigger basically the server trying to say, oh, I’ve been asked to format the string here, let me happily go and download whatever custom formatting utility you asked me to download and write it to the log file itself.
The combination of these situation is that the hackers have a trivially exploitable mechanism in which they are asked log something, and again, many servers just log a ton of user input as part of the normal operation, and then the logger would go ahead and reach out to the internet downloading a custom unverified code that pretends to be a formatter, basically allowing the hackers complete and unfettered code execution on the system.
So it’s a trivially exploitable bug that basically allows hackers to ask the servers politely, please reach out to the internet, download my custom code, and run it on the server that is currently vulnerable. It is a very severe vulnerability.

[Nate] You alluded to the severity of the vulnerability there.
This vulnerability was recently given a CVSS score, so before we talk about the score itself,
just tell me quickly, what is this CVSS scoring system for listeners who don’t know, and how
does it work?

[Yonatan] CVSS system is looking at each vulnerability and how easy it is to exploit, how impactful exploitation is, so how much damage will exploitation cause, what is the technical barrier or natural mitigations that would occur to limit the scope and spread of an attack, and then gives analysis in the number between 1 to 10 of how critical a vulnerability is. It’s considered that any vulnerability with a score of 7 or 7.5 and above is considered to be a critical vulnerability, where famously, dangerous acts of the past were given a score of 9.5 to 9.8.
So now for the big reveal, the MITRE organization gave the CVE the score of CVSS 10, the highest possible score you can get, which means it’s a very severe and very few mitigating factors and allows a vulnerable, complete code execution on victim servers in a ubiquitously available server. It doesn’t get worse than that in terms of vulnerabilities.

[Nate] Just to bring the point home a little bit, what does this 10 really mean? Because I’m looking outside and the world isn’t exactly on fire.

[Yonatan] 10 means that basically, it’s simple to exploit vulnerability with very few mitigating factor, with a very high prevalence of the risk, allowing us to do things like worming, which means a single server that is being attacked could then therefore go and attack other servers, so it’s wormable, is in a sense the most severe kind of vulnerability there is.
And to your comment, “the world is on fire”, well, naturally we’re not seeing the fire, but IT teams across the globe are frantically working right now on patching their systems and making sure that they are up and fighting against this vulnerability. At the same time, hackers across the globe have weaponized this very quickly.
We mentioned earlier how easy it is to exploit this vulnerability, which means that, basically, right now there are millions of scans happening all the time on literally every host trying to exploit common patterns on exploitation of this bug.

[Nate] Okay, so we know what kind of damage this vulnerability could lead to. Do we know of any damage that has already been caused as a result of people discovering this?

[Yonatan] There are clear traces of this being exploited at least a week past globally on very small scale. So before it became so widely known and every hacking group adopted this into their arsenal, there are clear signs of this being exploited. The researchers across the globe are communicating, seeing a huge influx of attacks. Basically, talking about tens of thousands of requests per second globally scanning the internet for vulnerable systems.
Interestingly, the first research exploitation of this vulnerability was actually against Minecraft servers and Minecraft players used to deploy cheats and a similar way to cheat in the Minecraft game or leverage that for access. So that was an interesting observation.
But then to your question, it’s very clear, at least initially, these will be hastily created payloads whose purpose is to assure future access. This is right now a land grab before defenders have a chance to patch their systems for hackers to go and make sure that they have beacons or backdoor systems in the way that they can come back in the future.
I’m sure we’re going to see a ton of leveraging of these attacks over the next couple of weeks and months happening. But definitely we are seeing companies getting breached left and right right now and this exploitation is so trivial that it has been weaponized very quickly.

[AD] The attack surface has never been larger or more diverse, yet defenders are still forced to piece together intelligence from numerous siloed solutions that produce a flood of alerts in order to detect and end complex malicious operations. No more.
Defenders can now leverage AI-driven Cybereason XDR powered by Google Chronicle to predict, understand and end sophisticated attacks with the only solution on the market that delivers planetary-scale protection that allows them to predict attacker behavior through a revolutionary, operation-centric detection and response approach.
Cybereason and Google Cloud are dedicated to teaming with defenders to end cyber attacks from endpoints to the enterprise to everywhere.
Learn more about Cybereason XDR powered by Google Chronicle at Cybereason.com slash platform slash XDR.

[Nate] So you, Yonatan, represent a cybersecurity company. You’re at the forefront of this. When you got the news of this vulnerability, what did you do? What occurred at Cybereason? Bring me there.

[Yonatan] So Friday morning, kind of the world at large gets to know of this vulnerability and being publicized and we realized quickly that this is a big issue. So naturally, as a software vendor, we have one track working very actively at this point. Are we ourselves at risk of this vulnerability?
Patching systems that need be, scanning everything, making sure that what the answer is, and blessedly through a series of software solutions and mitigations, we are not vulnerable. Great.
Second track of the security vendor for Cybereason, are our customer-protected? So making sure that our customers remain secure and our system provides for them the visibility and solution to find and remediate that threat.
But the third track, which became clear immediately, which was how do we help defenders globally, whether customers or not, be able to secure against it? At the time, we’re talking about around Friday morning, Eastern time, where the Apache website themselves are talking about a potential flag that you could define or either, of course, use the patch that they have just released or use a flag that was disabled by default on vulnerable servers, as long as you have a relatively recent version that could be used to disable this.
However, operating that flag required two things, A, it required admin access to the vulnerable servers, and B, it requires a server restart, which, of course, means downtime and all of those related things require for access. We realized that this might not be enough.
During a conversation between myself and my co-founder, Yossi, we started talking about when the vulnerability is so easy and runs within the Java system, we can actually use the vulnerability to go and just flip on that switch without restarting the server.
I liked the idea and reached out to one of our lead developers, Mayan Selam, and asked him over the weekend, do you want to explore that option? He, of course, says, I’m in, I’m all in, let’s go and explore that question. So we started looking at what does it mean for us to write a payload, basically an exploit,
so to speak, but use the vulnerability whose sole function is to turn on that switch that Apache documented without requiring a restart, without requiring access to the server, just allows you to flip the switch on on a server. And it took us basically two or three hours of work to get something that we saw was very stable, worked across all supported versions of Apache Log4j, and allowed us to go and basically vaccinate the server before using that vulnerability.
The nice thing was if you had used that our vaccine on a non-vulnerable server, nothing would happen. If the server itself was not vulnerable, you could not, quote unquote, reinfect it because it was not vulnerable. But if you had a server that was vulnerable, once you deploy the vaccine, it effectively works like the official mitigation, flips the switch and asks the logger to read the new definition with that switch set to disabled. This has been something we pushed, we decided to release that as an open source solution, put it on GitHub, make it public, and try to communicate to the world this is available. The answer was that it just works.
We’ve seen companies, therefore, coming to us and saying, we’ve adopted this solution as a way for us to mitigate the risk and basically buy time before applying the actual official patch. That’s something worth spending a few seconds on.
Our vaccine isn’t supposed to be a long-term solution. The Supposed long-term solution is making sure that you’re patching your software and making sure that you keep up to date with the system. However, with the rate of tens of thousands of exploitation attempts happening per second on the internet, the best thing we can do is allow defenders to buy time to go and patch their systems.
So instead of having a race against the clock where defenders need to go and patch their systems quickly enough, now they can say, we’ll use the vaccine, we’ll lock down our environment and then we can have time to go one by one and patch our systems, making sure of all the quality controls and mechanisms required to get back to operational state.
That was our purpose in building and releasing that vaccine as an open source solution.

[Nate] Is there anything that people or organizations can do to protect themselves from future zero days that are like this one? Any sort of general principles of security that might apply? Or is it that we don’t actually know what we don’t know?

[Yonatan] In general, this is a continuation of our cat and mouse game, which is, there will be more zero days in the future, hopefully nothing this severe in the next couple of months or at least a couple of years, but historically we’ve had one of those scale things every couple of years. Now we’ve had Athenium last year and we had the Log4Shell right now and hopefully we’ll have some quiet time here, but in a sense, these things will happen.
The strategy remains to be work under the assumption that the thing will happen and understand what will become your vulnerability surface. How do you use solutions that monitor for exploitation and are able to respond as this threat occurs? How do you establish the, on the one hand, the IT hygiene processes required for very critical patching available on one hand and on the other hand, make sure you have enough visibility, enough monitoring, enough hunting capabilities to understand that these things will happen and you have to be prepared for the time it does.
Naturally, solutions such as EDR, for example, our own Cybereason EDR, as well as a cloud of workload protection and system monitoring solutions are critical in order to be able to find these things as they happen, as they enroll in the future and now of course.

[Nate] I think you briefly alluded to this earlier. Is it possible that some malicious actors somewhere have already leveraged this exploit in such a way that they could have left back doors in organizations that we now have to go finding?
If so, how can we discover what may have been done already and possibly remediate it? I know that this is all very theoretical.

[Yonatan] Given the speed in which this caught on and what hackers will be doing, it’s very likely that they have indeed backdoored the system.
Now, the solution for this is the same as we discussed earlier. Be vigilant, assume that your systems may have been compromised, understand your patching strategy and understand how you’re monitoring your system for when things start behaving maliciously.
So having a behavioral monitoring solution such as an EDR is key to fight this as well as other similar exploitation in the future.

[Nate] Yonatan, how about a parting thought to take us out with?

[Yonatan] I think that the vaccine we developed really allows defenders to get control of the situation. It is not meant to be a long-term fix and long-term fix remains patching and updating your software or deploying one of the more official mitigations into your software development and into your live environment.
However, it is clear that for many years now, we’re going to have more and more unpatched systems, everything from vendors who are no longer in operation or have left their system unpatched through the ever-present threat of these things in IoT. The vaccine offers a viable approach of saying, if we have a system which we can’t go and patch, if we can’t go and log in to mitigate, we can use that to block the threat.
But I think if we look a little bit from a step up, what we’ve done here, weaponizing and exploits to close a vulnerability, so weaponizing a vulnerability to close a vulnerability is a very interesting paradigm. It allows and empowers defenders to say, if I have an issue in my environment, I can actually leverage and in a sense, flip the risk on its head, reverse the adversary advantage and use that risk in order to as quickly as possible deploy my remediation solution.
So the mere fact that this vulnerability is so trivial to exploit, it has a CVSS score of 10 and it’s so easy to weaponize, also means that for us, deploying and creating a very stable vaccine became easy, which is why we can do that over a span of a couple of hours, push it publicly and make it available for defenders all over the world to go and protect themselves.

[Nate] Yonatan, thank you.
Listeners, if you want your shot of the vaccine, visit cybereason.com/Logout4Shell-vaccine.
That’s cybereason.com/Logout4Shell-vaccine.
Thanks for listening.