Software Bill of Materials (SBOM) [ML B-side]

Transcription edited by SODA

[Allan] We need a shared vision of what basic software supply chain transparency looks like.

[Ran] Hi and welcome to Cybereason’s malicious life B-side. I’m Ran Levi.
Winston Churchill, who led Britain in World War II, is credited with saying, never let a good crisis go to waste. I’m pretty sure he wasn’t talking about cyber security, but you know, great insights into human behavior have a tendency to be relevant across a wide range of different domains.
Software Bill of Materials, or SBOM for short, is an idea that has been floating around for quite a few years. It is inspired by or derived from an established practice in the field of manufacturing, one which was old news even back in the early 2000s, when I was an electronics engineer. In hardware, a bill of materials is a list of all the different components and items, from capacitors and resistors to silicon chips, included in a product. A software bill of materials is a very similar idea, just for the software side of a system. It is a list, or a nested inventory, of all open-source and third-party software components present in a codebase. It lists all the versions of these software components, their patch status and even their licenses. This way, when a new version of a software component is released or a new vulnerability is discovered in some library, developers can easily identify which of their codebases are affected.
I think all of us can intuitively grasp how useful such a list could be. A typical electronic board has hundreds if not thousands of components, but when a vendor announced a recall of a leaky capacitor or a new version of some chip, all I had to do was a simple search in my excel files to see which of my projects used that capacitor or chip. Similarly, a big software project can include a large number of third-party libraries, and to make matters worse, each of these libraries can depend upon its own variety of other libraries, and so on and so on. An SBOM can really make life easier for developers.
Yet for some reason, I’ve never heard of SBOMs until recently. There was an attempt in 2014 to pass a law that requires government agencies to obtain SBOMs for any new products they purchase, but this legislation got nowhere, which is a shame, because such laws, even if pertaining to government agencies only, have a tendency to drive far-reaching changes in the commercial world as well.
But the past two years have been a time of crisis for cybersecurity. The infamous solar winds attack, which affected some 18,000 government and commercial organizations, was discovered in December of 2020. Six months later, we got the Colonial Pipeline ransom attack, and last December, researchers from Alibaba Cloud uncovered the Log4j vulnerability, one of the biggest, if not the biggest, vulnerability in recent history.
Apparently, the Biden administration listened to Churchill’s advice from 80 years ago, and in May 2021, following the solar winds and Colonial Pipeline attacks, the president signed a new executive order, asking the National Telecommunications and Information Administration Agency to publish guidelines about SBOMs, and forcing all government agencies to require SBOMs from their subcontractors. It is this executive order that finally brought this important idea of SBOMs to my attention, and now yours.
Nate Nelson, our senior producer, spoke with two SBOM experts, Allan Friedman, who leads the SBOM efforts at CISA, the U.S. Government Cybersecurity and Infrastructure Security Agency, and Chris Blask, vice president of strategy at Cybeats, which provides supply chain protection services and inventor of one of the early firewalls back in the 90s.
Enjoy the interview.

[Nate] Tell me about the state of cybersecurity before SBOM, so the kinds of problems that we were repeatedly facing.

[Allan] We don’t have to imagine what life is like without SBOM and what it would be like.
Last December, a lot of us, including most of your audience, I imagine, had a very busy holiday season, as we all had to scramble to figure out where we were affected by Log for J. There are a handful of organizations that had already implemented some measure of transparency to their supply chain that was thorough, using SBOM or related technology. They were able to say, yeah, let’s run some scans, let’s do some basic analysis of the data that we have, figure out where it was important and what wasn’t.
Everyone else actually had to do a lot of manual work, build some custom tools, yell at their suppliers to get some information, wait for CISA’s massive GitHub repository of every piece of software on the planet to update. It was a real pain.

[Chris] For me, it was like a lot of things. It develops over time.
One particular event, a CyberSend event in London in, I think, October 2018, and we had a panel with four of the top folks, ISA chairs and so forth, and somebody asked, we started the topic of supply chain, and just in that conversation, looking at it and saying, we don’t have anything, nothing whatsoever on supply chain.
Then in early 2019, February, with Tim Roxy and Guyana, a little second world country that suddenly ran into $100 billion worth of oil and had their national infrastructure hacked, and spending a week walking around looking at that and having to say to them that there is no way I or anyone can tell you or you can find out what is inside any of this equipment with all these foreign logos on it in your infrastructure.

[Allan] In the healthcare sector, especially after the WannaCry virus exposed the fact that many people just had no idea what was running these medical devices that were often internet enabled or internet connected, and so they simply didn’t know, hey, it’s the blinking box that’s keeping grandma alive going to be susceptible to a worm. The FDA said, hey, maybe we’re going to start regulating this.
Meanwhile, DOD, similar concerns about supply chain, especially for software that came from potential foreign influences or adversarial influences.
The automotive sector has always been interested in supply chain, and the real risk was that each community would develop their own solution, their own standards, when in fact, we all use the same software.
The first step was really to say, we need a shared vision of what basic software supply chain transparency looks like built in a way that it can enable a huge range of use cases.

[Nate] That leads me perfectly into the question, either of you are perfectly capable of answering it. What is SBOM?

[Chris] Allan, the floor is yours.

[Allan] The basics of an SBOM are quite simple.
It’s the dependency graph of software in a way that you can identify what’s in the supply chain. The analogy that we always use that’s imperfect, but still useful is a list of ingredients, and it’s nested.
So right, my cookie contains chocolate, chocolate in turn contains cocoa solids, and similarly, software is not hewn out of alabaster marble by tortured monks on Greek islands. We know that it’s assembled, and each of those subassemblies is interassembled.
So an SBOM is just the dependency graph with enough data to take information about my software components and map that to other sources of information that I care about. Now, there may be a lot of data that we want beyond that basics. So for example, some people care about licensing. Some people want to put vulnerability data in the SBOM, and we can talk about that because there are some risks there.
But at its core, the basic SBOM is just what are my dependencies, what are their dependencies, and all the way down.

[Nate] Okay. So basically, it’s demonstrating what software is in what software for the sake of supply chain security.

[Allan] While there are lots of use cases that are not explicitly around supply chain, for example, are there known vulnerabilities in my software? The data reflects the long supply chain that builds all modern technology today. We’re all in the supply chain. Most of us are in the middle.
So we take things from our upstream people, and we ultimately pass them downstream to further customers or users.

[Nate] And so it’s just good cyber hygiene to keep a track of all of this.

[Chris] What I think an SBOM is, is an inevitability about software inventory that is just time, right? With major events like firewalls, right?
They existed before the big firewall wave, but you need to look at these things and say, why would I not want to not have this? Why would I like to not have a defined boundary between me and this internet thing? And SIM, network monitoring, why would I like to not know what’s going on in my network? Threat intelligence, why would I like to not know what’s happening to other people? And this one, why would I like to not know what’s inside my software?
So now that SBOM is getting formalized and we’re getting to this stage, that’s the real question. Why on earth would you want to not know these things?

[Nate] So how common are SBOMs now? What percentage of vendors are already implementing them?

[Chris] If you said 1% right now, that’s probably optimistic, right? But it’s not about adopting something new.
People have the whole, anybody who knows what CICD pipeline means understands a lot of the steps already. And organizations are going through this process with or without a SBOM standard or a presidential executive order because they spend a lot of money writing and using code and they want to know where it is. And they want to spend less money writing it, managing it, and trying to figure out where it went.
By relative terms, it will still take quite a number of years to get to 100%.

[Allan] I’ll add that the administration has started to prioritize a better visibility into the open source community.
Now, again, that’s a massive ecosystem, but we’re working with folks like the Open Source Security Foundation to think about how do we push this metadata into the supply chain. Once that data is in place, it’ll be much easier for organizations to sort of pick it up and pull it in, building this data recursively rather than each organization having to do the basics.

[Chris] You know, when you think of the scale and dynamism of this, there’s an artificial intelligence bill material working group right now going through this. I think it’s a good thought exercise because we’re moving to a world where a lot of code is going to be created and compiled on the fly for particular purposes. But we’re thinking about the old world where I have four software releases a year, so that’s four SBOMs when it’s really 4,000 a second or whatever.

[Nate] Even if we agree that SBOM is a good thing, say I’m a vendor and I’ve never done something like this before, what is the cost, the challenge involved in creating an SBOM?
Is it simple or could it be like a big investment in terms of time and money that actually not everybody has the immediate resources for?

[Allan] So for a lot of organizations, this is going to be a one-time cost, either to get the basic metadata in place or as Chris mentioned, starting to use some of the modern software development tooling and processes, which is something that they probably want to shift towards anyway.

[Chris] Right. I’ll take this one with my commercial hat on.
So there’s a non-zero cost that’s related to, though, as I keep repeating in this, the things you’re doing already. And going forward, I see really quickly one of these competitive things where the world is changing. You’re not going to be carving software and clay tablets by hand anymore.
This can be integrated with in-toto attestations about custody of who’s touching us, the code, automatic SBOM generation of various places for various purposes, and so on and so forth. And if you don’t do that, you can do it by hand and you’ll have higher costs and slower customer responses and your competitors won’t.
So I think any actual speed bump startup cost is pretty well absorbed real short term in just being more efficient in the first place.

[Allan] And there are a couple of ways that we can also think about this.
One of them is just to sort of examine the status quo. What do you mean you don’t know what you’re selling your customers? They’ll say, well, okay, maybe we have some ability to understand what’s in our product. Okay, well then let’s talk about how do we automate that and how do we scale that.

[AD] The best strategy for organizations to avoid becoming a victim of ransomware is to prevent the attack from being successful in the first place. Cybereason remains undefeated in the fight against ransomware because it moved beyond alerting to deliver an operation-centric approach that detects and prevents ransomware attacks at the earliest stages of initial ingress and lateral movement. The Cybereason predictive response capability disrupts ransomware attacks prior to data exfiltration and long before the ransomware payload can be delivered.
Visit cybereason.com to learn more about predictive ransomware protection and how your organization can realize both increased efficiency and efficacy through an operation-centric approach to security operations.

[Allan] The other thing that I think is useful is a lot of security costs work against the scale of small businesses and startups, right?
Just you have to get a fixed certification cost. It’s much easier for the big guys. It’s much more expensive with small guys.
SBOM is one of those fun areas where the smaller and newer you are, the easier it is to do this because the greater the chance that you’ll be able to slot in existing tools in often open source tools.
Whereas a lot of the pain that we’ve been hearing about is from these very large organizations with dozens or hundreds of different product families, each of which is run separately, and all of a sudden these cylinders of excellence, they have to figure out how to talk across.

[Nate] So then why would bigger organizations go through all the trouble?

[Allan] The good news is these organizations that also realize the benefits of tracking this data. For example, we’ve talked to one industrial control system giant who said, hey, if we had SBOM today, it would save us thousands of hours a year because whenever a new vulnerability comes out, we need to quickly and easily see whether or not we’re affected.
How do we communicate that to our customers? The savings from doing that at scale is going to be huge.

[Nate] As two people who are invested in this, I imagine that you guys are working with or talking with companies that are dealing with SBOM maybe for the first time.
How often would you say this is a surprise and a challenge? How often do vendors really not know what they’re selling until they do something like this?

[Chris] Yeah. Having worn a lot of vendor hats over 30 years, it was not possible to know what was in your software until now. I mean, yes, you could have done it, but again, it’s not a matter of security, it’s a matter of economics.
Whether any of the small companies or major global corporations I’ve been with, if we had taken the time 10 years ago and 20 years ago to write down, get more people, more programmers, more engineers to document every single step and every open source library included, we could have done that. Our cost of our product would have been double our competitors. It just hasn’t been possible to get there.
We love to pick on the vendors, but vendors are made of individuals with jobs and if you can’t stay in business, you can’t do the right thing. We’re at this point now where, frankly, you can do the right thing from a security perspective, but more importantly, knowing where you’re putting your code and being able to find it, whether it’s a security incident or anything else, is just more efficient.

[Nate] A couple of logistical questions.
When and for whom are SBOMs legally required versus just incurred?

[Allan] I don’t want to speak too much to ongoing regulatory discussions that I can’t speak for other parts of the US government, but the executive order 14028 which came out late last spring, officially called for every agency that buys things, hint, US government buys a lot of things, the software they buys need to come with a bunch of basic security properties including you need to have an SBOM.
Right now, we’re finalizing what the actual enforcement of that looks like, but all the basic discussion on what you need to do, including when we say something’s an SBOM, what does the US government mean? That work has been completed.

[Chris] It’s the old paradigm of if we don’t do it ourselves, the adults are going to show up someday and force us to.
I would expect to see regulations in various sectors that are nation critical here in the US, in other countries as well, they won’t always be the same sectors, but if the market can’t regulate itself at some point, the state will.

[Allan] Chris alluded to an important thing that we occasionally forget here in the United States, which is there are actually at least four other countries in the world, maybe even more.
A lot of the work that we’re trying to do here at CISA is to help coordinate with those, make sure that we’re aware of this idea, they have buy-in for it, they can engage on their terms with it, but also at the same time, we want to make sure that we’re all moving in the same direction and that no one is trying to radically reinvent things in a different way. We don’t want to fork this effort and you can spell that with an R or with a U depending on your preferences.

[Chris] Well, I’ll forecast it, but I don’t expect the regulations in this area to be too high. We have enterprise benefits involved, when there’s some joint good we have to achieve that’s actually really going to cost people, that’s where you have regulations.

[Nate] Is there any evidence yet that SBOMs are already starting to improve things?

[Chris] Oh yeah, yes, and you look around, the sector’s OT has been overrepresented because of all the usual reasons and of the OT vendors involved, when log4j hit the Stuxnet of the supply chain, you see that the costs involved, you could just see it optically, to be clear, I don’t know the, I think I know, anyways, I don’t know everybody’s actual man hours and money spent in response, but the companies that were involved in Allen’s Groups, that had gone part of the way down the road on SBOM before the executive order, they recovered from that a lot faster, lots, a lot faster, saved a lot of money and a lot of time.

[Allan] Let’s take the company SolarWinds, famously was attacked along with a few others, which we disclosed in December of 2020.
That organization has done some great work rethinking their entire software development and employment process, thinking of it as a long multi-step process, trying to understand the risks of each step and figuring out how to secure them. SBOM is such a key part of that, we need to track our dependencies, but it is not the only part of it, it is a necessary, but not sufficient step to really understand what we need to do.
Some people say, hey, would SBOM have allowed us to detect or prevent the attack against SolarWinds?
No, I want to be very clear, it wouldn’t have had that impact, but all of the things we know we need to do start with visibility into our supply chain.

[Nate] Well, that’s most of what I felt I needed to ask.
Is there anything else that you two would like to leave listeners with?

[Chris] Two points I have is one, SBOM management is fascinating, working in Sybeats and seeing vendors actually taking all this data and starting to do things with it and connecting it out.
As Alan knows, don’t get me started, but once we start thinking past individual SBOMs and start thinking about the entire supply chain, we get to my second point, we don’t really talk about supply chain, we talk about transparency, because that’s what this is about. I believe fundamentally, I believe professionally, I believe philosophically that business is better with less overhead.
Transparency saves time, increases customer satisfaction, employee retention, everything. This little thing we’re doing inside, for all the grand landscape I’m trying to describe here, the fascinating thing about SBOM is it’s forcing us as individuals and organizations to say, oh, I know I already have a contract, I know we’re already doing the thing, but do I really want to write that down? Once you find that you do, and Alan knows, you’ve seen this case by case as vendors go through this process, they say, oh, that wasn’t so bad. It turns out, my customers are happy with me, life is better.

[Allan] Would love to share a little bit about what’s left to be done, because I don’t want to imply that this is a solved problem. The state of SBOM today is there’s absolutely no reason that organizations can’t start producing SBOMs from their software, using commercial or open source tools. They should be asking for SBOMs from their suppliers, they should be getting to think about what they could do with SBOMs when they have them.
But there’s still some important work to be done, and CISA is going to be playing an active role in facilitating that community-led effort.
We had a SBOMarama to talk about what the future of SBOM should be like in December. Four or Five hundred people participating virtually from around the world. We just wrapped up some listening sessions with a couple hundred people on specific topics, and we’re about to launch some work streams that are going to be community-led, facilitated by the government, but really driven by experts in the community.
Just very quickly, the four topics that we’ve identified are, one, what does SBOM mean for cloud and SaaS? A lot of our discussions are things like ICS, medical devices, automotive, where it’s on my network, I need to know what’s in it. But even those technologies now have online application side of it, and very few of us would say that the future of software is all on-prem, so we need to be able to tell better stories and think about what SBOM implementation looks like in those domains.
The second is on tooling and implementation. How do we make sure that there’s interoperability and scale? Because that is going to be key for having a thriving ecosystem.
The third one is going to be on how do we move SBOM data around? This isn’t unique to SBOM data. We need to have better stories for all the software supply and change data that we need. What’s going to be centralized? What’s going to be decentralized? What should be public? What needs to be access controlled? And be able to tell the stories and ideally pilot some technology that can scale.
And then the last piece is on the softer skill side, but is perhaps the most important, which is what are the on-ramps for adoption? How do we make it easier and cheaper for organizations to make progress in understanding what’s in their supply chain? How do we help the scale? And how do we track all of the different efforts that are moving in this direction around the world?
So, if any of your listeners are interested, more than happy to welcome them into the group. You can reach out to us, forgive the plug, sisa.gov/SBOM, or send us a note at sbom@sisa.dhs.gov.