As part of our Log4j Global Coverage Update presented by Celerium and its Ransomware Readiness Network, we recently did a video interview of Julius Musseau, CTO of MergeBase, a company focused on opensource libraries within software systems. Julius shared his thoughts on the Log4j vulnerability and how to handle it.
Celerium’s Chris Opp, director of SMB Cybersecurity Solutions, asked Julius Musseau several questions. Here is a recap of key questions from the video interview.
Tell us more about MergeBase. What does your company do?
Answer from Julius Musseau, CTO of MergeBase:
MergeBase is an SCA provider, software composition analysis, so we're really focused on opensource libraries within software systems, especially in the development phases, but we also cover production phases as well. So, we look at software systems and then we look at all the opensource libraries within them, and try to notify developers and users or operators of those systems of known vulnerabilities, serious known vulnerabilities within those opensource libraries.
Let's dive into the Log4j vulnerability. What is your opinion on the severity of this vulnerability?
Answer from Julius Musseau:
Yeah, extremely severe. It is an incredible bug. Because I'm in this business, I was actually in a way somewhat delighted to see the bug, because it's just a bug like this you just don't see them very often. The last time in the application security domain, so in the SCA in MergeBase's domain when we saw a bug like this was in 2017, the Apache Struts bug, which took down Equifax. This bug was pretty much at the same scale as that one. So, you'd expect to see bugs like this. Personally, I suspect every three or every two to five years. They're cataclysmic, incredible, amazing bugs.
So along with that, there's been many reports about the difficulty of finding instances of Log4j on computer systems. How difficult is it to find all the instances of Log4j?
Answer from Julius Musseau:
Yeah, I could really nerd out there and get super philosophical about that. So Log4j ... it's a very popular library, and it's a very old library. Its development of Log4j, I think started in 1999 and the first public release was in 2002. So, when you have a very self-contained, focused and useful library like that, Log4j itself was so successful that I think it was an inspiration for logging features that were later brought in native to the Java JDK. But the law for J itself remained a very powerful and focused and useful library.
So, when you have a small, useful solid library like that, and it's over 22 years old since it was publicly released, yeah, it's just going to proliferate massively, and any way that a developer might think to get it into their system, any way you can imagine bringing Log4j into your system, that's a way that someone who will have done it. So, that's a real challenge because the library's so old, and it's still actively maintained to this day; and because it's so useful, but also because it predates the way modern Java software development now brings in libraries. So, just a million different ways Log4j is being brought into systems, which means there's a million different ways you need to be aware, be able to detect it.
Your company MergeBase released a Log4j scanning tool. Can you tell us more about it?
Answer from Julius Musseau:
Yeah, sure. So, on the eve of the bug being announced, so the bug was first really started to reach public awareness around, it was December 8th or December 9th, so on the Thursday evening, late Thursday evening. And so by Friday morning of that week, we were looking into that and checking our own systems to see if our own systems were affected. And they actually were, we did have a vulnerable version of Log4j in our own platform.
And then, just based on some of my own academic background in software provenance and how to detect libraries on disk, I just thought, "Yeah, you know what? Let's whip up a tool, a little binary tool just to help companies find Log4j on their disk or inside JAR files inside JAR files or ZIP files inside ZIP files." And I quickly was looking on Twitter and online, seeing what other approaches people were taking. And I saw a lot of hash-based approaches, and based on my experience, hash-based approaches aren't very robust or resilient in the face of compiler variation. Like, for example, if you build Log4j from source yourself and you compile it with a Java 6 compiler or a Java 8 compiler or debug mode off or optimization mode on, all those things, you're going to get different hashes. So I just wanted to put together a simple scanning approach that would look for things that would be on disk that are not going to change in the face of compiler variation.
So initially, there was a great deal of discussion about the potential impacts of the Log4j vulnerability, and since this initial surge of information, there hasn't been a lot of news coming out about it, and the narrative seems to be shifting. Additionally, there hasn't even been very many major breaches being reported or other attacks that can be linked with this vulnerability. What is your view of the situation and the potential impacts?
Answer from Julius Musseau:
I think we're still at the wait-and-see stage, right? If you recall from 2017, the Apache Struts bug from 2017, the absolutely severe cataclysmic Apache Struts bug, was announced in March of 2017. And it wasn't actually until May, three months later, that Equifax was breached. There's a 96-page congressional report that goes into a lot of detail around how Equifax was breached. They had some old Solaris boxes, I believe on SPARCs, they weren't even on Intel CPUs, that just missed the patch.
They had a whole avalanche of unfortunate events. For example, they were running us scanner on those systems, but it was misconfigured, and so it was only scanning the root directory. It wasn't recursively scanning the entire tree. And so on. And so, you have just an absolute horrible bug like this with just horrible consequences, if you get breached, and then you have quite a complicated clean up situation. So I think it's really, I'm sad to say it, horribly, but it's just a matter of time. Someone out there is going to have an important system that's going to slip through all the cracks of all this patching effort.
So yeah, if you remember back to 2017, vulnerability announced in March, Equifax was breached in May. I think they didn't discover the vulnerability for a few weeks or months even, and then it didn't make the news until September. And so March to September, that's like six months. So, this Log4j vulnerability is December. So yeah, maybe a bad breach will hit the news in April, May, June of this year. I think with a bug like this, I honestly think it's pretty much inevitable that a few bad breaches will have happened.
Open source software and libraries have been a topic of security discussions for many years, but with the Log4j vulnerability discussion has been reinvigorated. What impact do you see the Log4j vulnerability having in this area?
Answer from Julius Musseau:
Yeah, right. Just a great, I guess, reminder for everyone. The way I think of it is like application security and known vulnerability in the application security space are a big vulnerability now. Like 10 years ago, 20 years ago, I mean, 20 years ago, I remember my printer would just start printing rude things that I did not ask it to print. That was sort of the beginning of this whole cybersecurity awareness for me. It was like, "Wow, my printer's just printing things." And so I learned, "Oh, I need a firewall." I didn't know that I needed a firewall at home back then for my home network and so on. And so this continual evolution of security getting better, but as security gets better, there's still different holes that we then need to plug.
It's like, "Okay, well, we've got the firewalls up, we've locked the doors. Oh, now we have to work on the windows. Okay. We've addressed the windows, now we have to address the chimneys." So, application security, it's like, so all websites these days or applications, they're all listening on HTTPS, port 443. So, I kind of feel like port 443 is like the application security port. And if your application is going to respond to user input or users doing stuff, then the attackers can get in through there as well. The Log4j issue is exactly on that theme. Someone sends a chat message or a forum message and boom, they take over your system. And so, yeah, it's a great bug for raising awareness about this kind of the last leg of this security journey, the applications themselves, they can be compelled to do bad things if you've got some vulnerabilities in your code there.
So, with this higher adoption, what steps can a company take to prepare or to mitigate against these kind of supply chain vulnerabilities?
Answer from Julius Musseau:
So, the Log4j thing had me thinking like it's a lot like earthquake preparedness. The cataclysmic bugs are going to come, and you need to be able to respond very quickly when they happen. So, you're talking like, can I get my systems patched? Can I get these vulnerable versions of Log4j out of my systems within 48 hours or one week or two weeks? You need to be able to respond very quickly to cataclysmic-level events like this. That's I think the main thing we're learning as an industry from this. And so that means you'll see stuff like s-bombs, software bill of materials, just good inventories of all your systems and what they're built from just becoming more and more important so that you can respond as quickly as possible when just absolute cataclysmic vulnerabilities like this are discovered and published.