Special: The SolarWinds Hack

Transcription edited by Ayo Joshua Tayo-Balogun

Hi and welcome to Malicious Life in collaboration with Cybereason, I’m Ran Levi.
Just a few days ago, news broke of one of the biggest and most significant cyber security incidents in recent history.
A breach into several federal organizations’ networks, at least one security firm, FireEye, and potentially up to 18,000 businesses in the US, including many Fortune 500 companies. The attack is currently, informally, attributed to APT29, aka Cozy Bear, a Russian state-sponsored hacking team.
This is obviously big news, and so we decided to invite Israel Barak, Cyber reasons CISO, to discuss this incident in detail.
Some of you might remember Israel from previous episodes of Malicious Life, including our last live event from a few months ago.
For those who don’t, let me introduce you properly.
Israel Barak, Cyber reasons CISO, is a cyber defense and warfare expert with extensive background working for the government where he established and operated various cyber warfare teams. Israel spent years training, guiding, and professionally mentoring new personnel, providing in-depth cyber expertise as it relates to cyber warfare, cyber security, and threat actors’ tactics and procedures.
Israel is also a regular speaker at leading cyber security industry conferences and events.

[Ran] Israel, thank you very much for joining us in such a short notice.

[Israel] Thank you, Ran. It’s a pleasure to join you again, as always.

[Ran] So, for starters, take us through the events chronologically, as they unfolded. It started with FireEye, right?

[Israel] It did, but actually, you know, the timeline actually started even before then. I think when you look at the events chronologically as they unfolded, today, we know that the operational preparation on the attacker side, and specifically setting up some of the command and control infrastructure, began at least as early as August 6, 2019, when some of the command and control domains were registered. And the first known versions of the SolarWind software modification was actually from October 26 of 2019. So as far as August of 2019, when they began that preparation. So based on what we know today, that operational infrastructure for this attack was actually built between August 19 and February of 2020.

[Ran] When is the first time we learn anything about it?

[Israel] So the attackers actually began to distribute that malware, right? Using that SolarWinds Orion software update process in March of 2020. And they did that for at least seven months until September of 2020. And according to some reports, actually perhaps even until November of 2020, they continued distributing that malware through the SolarWinds Orion software update process. Now on December 6 of 2020, FireEye disclosed that a breach into their own network had occurred that impacted their Red Team tools. Now at the time, that report did not include any explicit reference to SolarWinds. It actually raised quite a, I would say quite a bit of questions like, you know, why would a sophisticated adversary want to break into a network to steal commercially available software tools and know-how, which was the FireEye Red Teaming tools. And we didn’t have to wait long, I think, until the bigger picture started to unfold.
Now on December 13 of 2020, a week after the initial FireEye breach report, Chris Bing from Reuters broke the story that the US Treasury Department had been compromised by a sophisticated adversary. On the same day, December 13, FireEye disclosed a second report, and this time it was the SolarWinds report that tied SolarWinds into the breach that FireEye had into their network. Shortly after, Alan Nakashima from the Washington Post confirmed with background sources a couple of things. And number one, that the Treasury Department breach was perpetrated by the same group that
targeted FireEye. Number two, that SolarWinds was involved in both breaches. And number three was that the threat group was believed to be APT-29 or otherwise known as Cozy Bear or the Russian SVR.
Now also on December 13 of 2020, DHS and CISA issued an emergency directive to US government agencies to mitigate the compromise of SolarWinds Orion Network management products. And two days later, on December 15, a partnership of different organizations started taking down the known command and control infrastructure of Sunburst, which is the primary backdoor that was used in the attack. Now last thing, I would say is that I think it’s important to mention that this attack is not over and we’re still in an evolving situation, and I think we will know more over time. For example, what additional post-breach techniques were used by the attacker, or in other words, what were they doing inside those breach networks? We’ll probably know more about more accurate attribution of who’s behind this and whether other actors.

[Ran] Yeah, Trump famously said that might be the Chinese, the only one who claims that.

[Israel] Yes, but I think it’s the operation itself, as you mentioned, with the number of targets and the breadth and depth of this operation, as you can imagine, is a very, very complicated operation to operate over time. We might learn that other threat actors were playing one part or another in that as well. I think we’ll probably also learn more about the ripple effect of this massive attack, exactly who was targeted among these 18,000 potentially that downloaded that malware, who was impacted and how. So it’s an evolving situation.

[Ran] That needs explanation. We already mentioned a few times the name SolarWind, and I think that’s the key part in understanding how could one single attack cripple, or potentially cripple or breach so many organizations and departments in the US government, etc. So what’s SolarWinds and how does it relate to this attack?

[Israel] SolarWinds is a software company that primarily deals with systems and management tools that are used by IT professionals. Perhaps the most widely deployed SolarWinds product is Orion, which is a network management system or short NMS. Now, the Orion NMS has broad capabilities from monitoring and managing systems at data centers, servers, workstations, and network devices, etc., etc. Now, not every organization will have SolarWinds configured identically, but when they do have SolarWinds Orion configured with these monitoring and management capabilities, it can be a very effective targeting point for attackers to go after. And one of the reasons for this is that in order for this system to monitor and manage the network, Orion needs to have a very wide and often very privileged access across the network, SNMP, WMI, sometimes with agent installation on managed systems. And so that broad access makes an NMS system a very lucrative target for attackers. So I think the fact that attackers targeted an NMS system is certainly not surprising. And by the way, this is a scenario that we also encounter often in red team or adversary emulation exercises. But when you target an NMS system that gives you that broad access to the network, now, when we think about the number of organizations that were impacted, we need to ask ourselves who uses SolarWinds. And I think better yet, we probably need to ask ourselves who doesn’t use SolarWinds. SolarWinds is one of the most prevalent network management systems out there. Over 300,000 customers, and among them, I think you’ll see some of the largest brands, over 80% of the Fortune 500 companies, for example, federal government agencies. And I think that gives us a good idea of how far and how deeply the attackers may have gotten through this attack.

[Ran] Amazing. Now, people have been referring to this hack as a supply chain attack. What is supply chain attack?

[Israel] So a supply chain cyber attack is a type of cyber attack that seeks to damage an organization by targeting less secure elements in a supply chain. Or if you think about it in ancient times, a technique that was used to target a fortified enemy would be to poison their water supply, which I think is a good example of a supply chain attack.
And in a supply chain attack, it basically enables an attacker to gain that trusted access into organizations by hiding itself in a legitimate tool that the organization is receiving from a trusted third party supplier. And I think as organizations become better in doing their cyber hygiene, I think supply chain attacks remain a very effective vector to gain that trusted access in a way that can be very, very difficult to detect. Now, second, another advantage is that it can grant the attacker access into a very large set of targets, like we’re seeing with SolarWinds, with one effort. In this case, potentially accessing the entire customer base of the compromised supplier. Now the challenges with that type of operation are oftentimes operational security. Since multiple organizations can be impacted by that attack and it can be very widespread, you need to be fairly sophisticated in your operation to make sure that it doesn’t get exposed fairly quickly because of the sheer volume that you’re distributing that attack to. What that often translates to is a required level of sophistication on the threat actor side in terms of how to manage operational security during a very large scale operation.

[Ran] How prevalent are these supply chain attacks in general? Are they used often against enterprises?

[Israel] So supply chain cyber attacks are not new. We’ve been seeing between two and three, I would say, more significant supply chain attacks each year in the past several years.
We all probably remember the NotPetya attack from several years ago in which a Russian threat actor was able to cause massive disruption to a large number of organizations using a supply chain attack that included spreading a malicious software update from a Ukrainian software provider. But I think the attack on and through SolarWinds is likely the widest and very likely the most far-reaching supply chain cyber attack that we’ve seen to date.

[Ran] Is there anything that a company like, I mean, so many companies and organizations that use SolarWinds, could they have done anything basically? What kind of influence does an organization have on an organization like SolarWinds? Or is there anything else that the client can do to protect itself from such a supply chain attack?

[Israel] So I think tactically speaking, there are several things organizations could have done to protect themselves against this specific attack. First, the security of the NMS system is probably something security teams need to spend more time thinking about. In many organizations, and I dare say in most organizations, the IT operations team is deploying and configuring the NMS system, often without much involvement or guidance from the security team. And this is probably a tactical lesson that we have here.
And by the way, I think this also applies to security orchestration and automation or SOAR platforms that many organizations have been deploying that have far-reaching access and privileges that may need to be reviewed with additional scrutiny now.

[Ran] But that can press you stronger on this point. I mean, what if I was, for example, an IT manager in such an organization, what can I know about a third-party product such as SolarWinds Orion platform that would enable me to make such a decision or protect myself? Isn’t it like a black box that I’m buying and deploying on my platform?

[Israel] And you’re absolutely right.
You’re absolutely right.
I think that’s part of what makes detecting that type of attack and protecting against that type of attack so difficult.
It is that level of trust and the built-in limitation in how much you can validate the level of security and integrity of a third-party product that you’re consuming and putting in your network, which is why I think oftentimes when you talk about threat modeling of how we manage risk in these types of environments, the question becomes, how do we reduce the potential impact of a supply chain attack?
How do we make sure that we can contain it if it does happen?
How do we give it the least amount of privileges?
In most organizations, NMS systems are configured with way over-provisioned privileges. They can do a lot more than what they actually need to do.
So reducing privileges, containing their ability to access network assets that they don’t need to access, segregation, blocking their ability to access external networks or to communicate with the internet if they don’t need to, and if they do need to, then limiting or restricting the specific addresses that they could communicate with outside to only those that are needed. So I think when we talk about threat modeling of NMS systems, it’s often about the over-provisioning of capabilities that we see in most organizations that I think can lead to a very high impact of a supply chain attack like that.

[Ran] You know, I think also people are now thinking about how have we gotten to the point where one software system, Orion’s update SolarWinds system, is deployed in so many organizations and so many important organizations that by itself, I think, is a problem.
It’s kind of like a single point of failure for so many companies, isn’t it?

[Israel] Well I think at the end of the day what you’ll see is that products that are valuable, high quality and solve real problems proliferate through the market and you’re right, SolarWinds is, how shall I call it, it’s the SolarWinds is for NMS systems, like Kleenex is for tissues.
It is a very, very prevalent friend.

[Ran] And that could be also one major lesson from this incident.
When people decide on buying, purchasing a system, software, platform, whatever, perhaps there’s value in not following the mainstream decisions of the market. For example, it kind of reminds me of the problem with Windows platforms as being, you know, target of so many hackers and viruses and such, while Linux distributions, because they are less prevalent in the market, they are much less targeted.
Diversification is probably one thing that you need to also consider when purchasing something, right?

[Israel] I fully agree and I think when you look at more mature security organizations and more sophisticated security buyers, diversification of security products specifically is something that is built into their culture.
For example, they would not buy all their firewalls from the same vendor, right?
They will not buy all their VPN systems from the same vendor because they assume that any single vendor can be compromised and the supply chain attack can trickle down and impact their own networks.
And so they want to make sure that they’re building layers that can help them reduce the impact of that type of incident.

[Ran] Kind of reminds me of one of the most probably basic principles in nature overall, that if the organisms in the community are homogeneous, they are more at risk of being attacked by one predator or virus or whatever.
So in nature, organisms have kind of evolved to be heterogeneous, even sexual reproduction is kind of the same, it’s nature’s answer to these kinds of threats. So we need to take maybe a lesson here.
So Israel, what do we know about the sunburst malware so far from the preliminary analysis that people are already doing?

[Israel] So I think we know a couple of things.
First, we already know that the malware was deployed as part of a software update from
SolarWinds own servers and was digitally signed by a valid digital certificate with their
name and that really strongly points to a supply chain attack.
Now SolarWinds, because they’re a publicly traded company, published some limited information through the SEC that they believe the attacker did not directly compromise their code base, but rather compromised their build environment, which is where the code gets turned into packaged software that can be delivered to consumers.
I think we can also see the sophistication of the attacker here, both on the development side and on the operational side. On the development side, the techniques used as part of the development of the tool set, I think includes several very effective anti-analysis countermeasures that are specifically effective in a supply chain attack.
The first one was delayed execution. Essentially the backdoor checks that the product update has been deployed for between 12 and 14 days before it tries to beacon or communicate with its command and control. The reason it’s effective is because it is able to bypass the victim’s ability to secure their change management process. Right, a lot of organizations first deploy an update into a staging environment, then they check that it’s acting normal, and then they roll this out into production. But no one’s waiting for two weeks before they make those changes.
Usually a product stays in a staging environment for a couple of hours, maybe a couple of days.
But the fact that they waited for two weeks means that they wanted to explicitly bypass that change management process and basically preventing not just the change management process or that pre-deployment testing process from identifying that abnormal change, but
also for preventing the use of malware sandboxes and other instrumented environments from seeing it.

[Ran] The clients themselves can’t really, after two weeks, they can’t really make the connection
between a certain update to a third-party system to such and such behavior that is covered.
It’s not an immediate connection, I’m guessing.

[Israel] Exactly. A lot of these clients, I would imagine, a lot of these Orion users, even if they saw something happening, they very likely discarded it as this is just a legitimate Orion behavior. The attacker also included some anti-sandboxing behaviors.
Basically, what they wanted to make sure is they wanted to make sure that the machine that they’re deployed in is connected to a relevant domain.
If it is not connected to a relevant domain, the malware essentially will not execute. They also made sure that certain domain names can be resolved to public IP addresses and not private IP addresses, which is a technique that they used to evade sandboxing tools.
A lot of sandboxing tools, when you put a piece of software in it and you detonate it, they would reroute the traffic so they can inspect it into private IP addresses. By checking that certain domains are resolved to public IP addresses, they’re able to bypass these pre-deployment tests.
Those are techniques that are very, very relevant, not just for anti-forensics, but they’re also very, very relevant when you target specifically a supply chain attack. On the operational side, I think we can see the sophistication of the attacker in the way they set up and maintain separate sets of indicators of compromise for each target across a network of thousands and thousands of targets to basically reduce the likelihood for detection of this attack and make it harder for victims to investigate it.
I think another interesting angle from a technical perspective, also from a threat intelligence perspective is that potential connection to the VMware vulnerability.

[Ran] For our listeners, the NSA published an advisory regarding vulnerability in VMware that was
being actively exploited, as they termed it, by Russian state-sponsored attackers.

[Israel] Right.
I think one of the interesting correlations there is that if you look at the context for that NSA advisory and the vulnerability information, it basically talks about a vulnerability in five VMware products. And according to the NSA advisory, it was identified as being actively exploited by several Russian state-sponsored actors.
Now, the vulnerability in the VMware product or exploitation of that vulnerability essentially allows attackers to deploy a web shell on a system and gain access to protected data. But this vulnerability is a post-breach vulnerability only.
That means that it can only be exploited by someone who has already authenticated to the system and is already inside the network.
It is assumed today that this vulnerability was one of the tools in the tool bulk of the SolarWinds threat actor that was used post-breach after they were able to gain that initial access via that SolarWinds compromise, so they can further expand their level of access and control over victims that use these vulnerable versions of VMware. I think the level of sophistication, the amount of effort that was invested in effectively operating a very, very large-scale operation, maintaining operational security across hundreds and potentially thousands of different targets that are part of the same campaign and the same operation, it very much points to an adverse…
We often say the word sophisticated or advanced adversary, right? But I think this is the poster child of an advanced adversary and how they’re able to operate in those scenarios.

[Ran] One more interesting question that arises from this incident.
As you mentioned, Microsoft last week took over a domain used for command and control of the malware.
The malware itself is called Sunburst and there’s a certain domain that the code references and calls for commands.
Why should Microsoft be the one to take those kinds of steps? Aren’t the FBI or some other more formal organization, state organization, that should be taking care of such things?

[Israel] I think one of the areas of actual, I would say, one of the bright areas that we’re seeing here, one of the areas from optimism that I’m actually seeing in this particular incident, the SolarWinds incident, is how it’s not just one single vendor, but the entire community comes together to create a more efficient process. In past times, vendors would wait for a court order, for a regulator instruction to perform a certain action. Today, I think the industry has matured enough for different players. Very likely, this isn’t done by a single vendor in this space because one single vendor can have a limited effect.
As big as they are, they can have a limited effect over a global situation like this.
It’s about the community coming together.
It’s about organizations that are vendors, that are end users, that are practitioners, that are regulators in this space coming together and acting in the most efficient way possible to try to end a situation, not waiting for bureaucratic steps, not waiting until they have to, but trying to do the right thing together.

[Ran] I can see your point, and efficiency is probably a very good advantage of this kind of thinking. But then again, I have the feeling that I don’t know if I want entities, commercial entities such as Microsoft, taking decisions like taking down domains because they think these domains should be shut down.
On some level, it kind of troubles me that we give commercial entities like Microsoft such huge powers, authority over our lives.
Maybe I’m overreacting.

[Israel] No, I think you’re right, but I also think you mentioned and you framed it very accurately earlier by calling this a natural evolution and natural selection process that we’re going through. In a natural selection process, you don’t always get to the happy medium, the best possible option immediately.
You go from one edge to another edge and you sort of fluctuate between those options until you converge into something or an option that is the best fit to its environment. And I think we’re going through a similar situation in how we evolve as a security community.
We go from a past state where we were slow to respond, acting as single entities as opposed to a partnership, and we’re going into, we’re fluctuating between these different states and eventually we’re going to converge right into a stable state that’s going to be the most efficient state possible considering a certain environment.

[Ran] Yeah, very interesting to think about it.
Besides, before we part ways, any last lessons or thoughts that you have in mind following this incident?

[Israel] Well, I think on the more strategic side, I think we need to become better at managing the risk from supply chain attacks because they will continue.
We see how successful they are. We just saw how difficult it is to detect them and to actually take effective action.
Even though we now all know about solar winds, this incident is very much still ongoing and the attacker is still very much on top of what’s going on here.
We can see how difficult it is to detect and take this down and obviously we’ll see more of this. I think we need to ask ourselves a number of questions and specifically from a security perspective, we need to ask ourselves how we can prevent, detect and effectively respond to a supply chain attack and both after it’s discovered, like we’re seeing now with the solar winds incident, but more importantly before it’s discovered.
It’s switching from a reactive position where we rely on finding these indicators of a known threat in our network, that reliance on IOCs as they’re referred to, which is often not relevant for prevention, only for that post-mortem analysis.
I think we need to switch to using better behavior analytics and indicators of behavior to identify those subtle attacks through those chains of behaviors that they’re exhibiting in our environments. I think some examples for indicators of behavior from the solar winds incident is how they leverage domain generation algorithms to establish command and control communication.
How they ended up creating a situation where the solar winds application, the legitimate application started doing execution and persistence activities on those machines, those Orion machines that were very abnormal behaviors, but a lot of organizations did not have the tools or the analytics or the mindset to see that these subtle changes are happening in their networks and associate them with malicious activity.
I think we need to become better at identifying these more subtle behavioral signals and making sense of them through behavioral analytics. I think at the same time, we need to think about the data retention on our EDR and other monitoring solutions for historical context, for example, in this incident.
We needed to have historical context that goes all the way back to March for organizations to understand what the full impact of the incident is on them.
I think the bottom line is probably that the incident, I believe, calls for that additional security attention to threat modeling of supply chain attack vectors as well as innovation of a security stack with using indicators of behavior for detection of malicious activity via EDR and other monitoring solutions.
I think that proves to be a very effective proactive detection strategy against the supply chain as well as other subtle cyber attack tactics.

[Ran] Thank you very much, Israel.
It’s been a fascinating conversation that took us from cyber security to natural selection and evolution and all the way back to cyber security again.
It’s been a pleasure. Thank you.

[Israel] Thanks so much for having me, Ran.