SBOM: Good Intentions, Bad Analogies, and Ugly Outcomes

The problem is real and everyone means well. But poor metaphors are leading to proposals that do not fit reality. In the end, scarce security resources will be diverted to produce compliance make-work with no tangible benefit.

A Modest Proposal

Imagine the following scenario. The automotive industry has a problem. The manufacturers are not doing a great job of reliably acting on part recalls issued by the National Highway Traffic Safety Administration (NHTSA) and consumers are left to drive cars with potentially dangerous parts. Understandably frustrated, the government proposes an obvious solution: the dealers will be required to provide a full bill of materials for every car they sell. The customers can then periodically cross-reference it with NHTSA recall data to determine when they need to contact the dealer to replace a recalled part.

Not only is this a roundabout way of addressing the problem, it actually makes things worse. It explicitly removes the responsibility for supplying recall information from the manufacturer and makes it the consumer’s problem. Which is the exact opposite of the desired outcome.

That is SBOM in a nutshell. Or it could be, if not for all its other flaws.

Good Intentions

Behind the desire for something like a Software Bill of Materials (SBOM)[1] is a very real problem: users of software products (either application software, device software, or SaaS) do not always get timely and accurate vulnerability information from their software providers. This applies especially to vulnerabilities that may be present in software components that were developed by third parties and integrated by the final product provider.

The basic idea behind SBOM is that if every software product lists all of its constituent parts, then the users themselves can infer applicable vulnerability information by joining SBOM and CVE data. As illustrated above, this is an unnecessarily circuitous and ineffective way of pursuing the desired outcome.

Bad Analogies

I do not dispute or minimize the underlying problem. But the proposed solution is misguided. The allure of SBOM is based on flawed metaphors and circular logic.

Components are not atomic

The first poor analogy is that of the “bill of materials” itself. The notion that software can be decomposed into discrete component parts, like a car, is appealing, but wrong. That is not how software composition works. A developer “using” a third party package may incorporate it in its entirety, or select individual files, or functions, or even lines of code. They may include these components as is, or further modify them. There is no atomic unit of software. Such fractional pieces do not have their own identifiers or versions and are not possible to meaningfully represent in a bill of materials. These are not rare edge cases. This type of software composition is common in many environments and is where the real difficulty of tracking third-party vulnerabilities lies.

Vulnerability is contextual

The second shortcoming of the metaphor is treating vulnerability as an inherent and transitive property of a component — if a component is vulnerable, then so is every system that incorporates it. Reality is, once again, much more complicated. Whether the compound system is vulnerable (and with what severity) depends on how the component is used. A tire may have a fatal flaw that results in blowouts at high speeds. But that’s irrelevant if that tire is used as a tree swing. A vulnerability in a cryptographic library can be a critical issue. But if that library is used to shuffle cards in a single-player solitaire game, then it may not matter. To assess applicability and impact of the vulnerability on the overall system the customer still has to rely on the supplier, who is the only one that knows how the component is used[2].

“in-depth knowledge of the embedding software [is necessary] to determine what action is appropriate to remediate or mitigate the effects of a vulnerability in a component.” - SAFECode

Layers on indirection do not eliminate fundamental dependency

Which brings us to the Escherian logic at the core of the SBOM mandates. They attempt to solve the problem of not being able to rely on a vendor to provide information of one type by relying on them to provide information of another type. The original problem is lack of accurate vulnerability information on third-party components from software providers. And we are solving this by requiring those same software providers to supply low-precision, high-noise intermediate data that can be used by consumers to later request the information that they needed in the first place — third-party component vulnerability assessments — from the very same providers that could not be relied upon to produce it in the first place.

Ugly Outcomes

All of my skepticism could be dismissed as nihilistic pontification with a simple existence proof. And perhaps some day it will be. But for now, the benefits of SBOM remain purely hypothetical. Specific instances where SBOM would have helped are hard to point to.

“Recent high-profile supply chain attacks (e.g. Sunburst) would not have been detected or prevented with an SBOM.” - Dell

Best [idea but not in] practice

Perhaps the greatest indictment of SBOM is the absence of large-scale working examples. That is the final flaw of the SBOM proposals: they are based almost entirely on thought exercises. Most industry "best practices" are based on actual practices — working systems and processes that evolved through years of effort and refinement, proven in real world deployments. In sharp contrast, SBOM is being recommended as a best practice in spite of it not actually being practiced. It’s an idea so good that nobody has bothered to make it work.

Much of the text in the NTIA request for comments describes benefits from an SBOM that are highly speculative. It is important that no guidance or requirements about SBOM be issued under the Executive Order until there are clear and complete demonstrations at scale that the putative benefits are in fact realizable.” - SAFECode

The best need not apply

Ironically, and noteworthy for that reason, a customer’s desire for SBOM is inversely proportional to the security maturity of a product. To the closest approximation, nobody is planning on devoting resources to monitoring the SBOM for Chrome, or Windows, or iOS. People really want SBOM from suppliers that are not already trusted to provide timely patches. It’s a best practice that the best are not expected to practice.

And so SBOM becomes a pointless compliance exercise for the trusted providers, wasting time producing something that will be universally discarded. And of course there is no reason to believe that untrusted providers will treat it as anything else either.

The writing is on the wall

While all of this venting may have some cathartic benefits, I realize it is unlikely to slow down the growing momentum behind SBOM. The requirements will come, and suppliers and consumer alike will divert already scarce security resources from efforts that work into compliance paper shuffling that somebody thought should work.

Cross-posted on Medium

P.S. Don’t take my word for it

I’ve included an Appendix (below) of some excerpts from public comments received by NTIA. Many of these are from large corporations, and it can be tempting to dismiss them as self-serving (though large corporations generally favor burdensome compliance regimes that make it harder for smaller entities to compete). Others are harder to write-off as corporate shills. They are names like Katie Moussouris (Luta Security) and Steve Lippner (SAFECode).

Notes

1. In discussing SBOM with other people in the industry, it became apparent that it means many different things to different people. In the most generic interpretations, SBOM is any metadata associated with a build. For example, to some, a digital signature on a build is SBOM. That’s not the SBOM I discuss here. My concerns are with a narrower, literal interpretation of SBOM as the software bill of materials — a detailed listing of all software components in a software deliverable.
2. All of this does not mean that suppliers cannot, or should not, track their third-party dependencies and associated vulnerabilities. And mature suppliers already do. But there’s a huge difference between having that information for in-house consumption by developers and exporting it to customers. What the above sections show is that this information can only be meaningful to the developers.

Appendix

Unashamedly biased selection (emphasis is mine) from public comments received by NTIA.

“Monitoring SBOM vulnerabilities should be up to the entity receiving the software, and that should be made clear to the receiver. […] Monitoring for vulnerabilities can be costly on an ongoing basis but a supplier could be contracted to provide such service as part of a contractual arrangement. ” - Arm

“An SBOM, if properly tailored and aligned with international standards, can provide useful information to customers, but if too prescriptive or not understood as a smaller part of a larger cybersecurity risk management program, can offer misleading information or simply not be a useful security investment. […] Detecting vulnerabilities through an SBOM requires mapping component identity fields to existing vulnerability databases. This mapping will not, however, explain if the component is used in a manner that would allow the vulnerability to be exploited. An SBOM is likely to cause a US agency to reprioritize its actions, in response to a vulnerability identified through an SBOM even if the software is not exploitable as used. If required to expend resources to remediate the identified non-exploitable vulnerability, then that agency cannot use those resources on cybersecurity activities that it would otherwise, correctly, identify as higher priority. […] Requiring a company to publish an SBOM, especially when software is not managed by the customer, would not improve a customer’s cybersecurity risk management, and may unintentionally assist adversaries.” - BSA (The Software Alliance)

“focus on SBOM as a quantum leap forward in maturity in third-party component management may, however, be misplaced. Recent high-profile supply chain attacks (e.g. Sunburst) would not have been detected or prevented with an SBOM. […] This information, without additional context (e.g., back-ports of security fixes, compensating controls, or even component modification) can be misleading. Providing version information in some cases will add to purchaser confusion and supplier overhead. For example, a vendor may have customized a specific open-source component to fit its needs, as allowed by the license. Later, the vendor may backport a vulnerability fix to avoid having to customize a later version of the component. Describing these modifications in a manner which will educate purchasers, and not confuse them, may not be feasible. […] Any requirement that would require significant R&D and Customer Support investment could realistically come at the cost of other quality or security measures that may ultimately have a greater mitigating impact on the risks posed to purchasers.” - Dell

“There might be cases where a software component that is used by a product and included in its SBOM is identified as vulnerable, but this vulnerability might not make the overall product vulnerable, In such scenarios, SBOM might give a false signal about the vulnerability of the product. Vulnerability ratings can be use case dependent. […] Thus, the level of details related to the security criticality of software components in specific products and the control of SBOM data needs to be identified and managed by the organization developing the final product.” - Ericsson

“Software identity is a huge challenge today. The most viable solution today is to allow SBOMs to record multiple ways to identify the same component […] Closed source software packaged as binaries for distribution is often best identified using a strong cryptographic hash. Open source software that can be compiled by anyone is often best identified using the URL or ecosystem-specific coordinates for the source release, not just cryptographic hashes, since recompilation and reuse of version numbers for slightly different versions of software can occur.” - Linux Foundation

“the concept behind SBOM is laudable, though, in its current inception, its introduction may divert time and effort from other more immediately beneficial cybersecurity measures and initiatives, which in turn can cause more issues for national security than it solves. […]

The assertions made by the proponents of SBOM, including the case studies published by the NTIA working groups, do not contain any data proving their assertions. […] The presence of an SBOM has no tangible or measurable effect supported by data in NTIA’s own case studies […] These “case studies” are devoid of data for their stated assertions, and more importantly, leave out data on the people, process, and technology that is required to understand the true cost and mechanisms by which SBOMs can make a tangible speed difference. While the concept of SBOMs is useful, it cannot be mistaken for a way to address software supply chain risk management on its own. There are unintended consequences of imposing an undefined, non-standardized, and unproven mechanism upon federal contractors. As we work to elevate the security of all members of the supply chain, SBOM production and consumption can be a potentially costly distraction from more directly useful activities. […]

An ingredient list of software alone is not useful to determine risk quickly without additional analysis by skilled workers.[ …] This is because from a technical standpoint, a bug in a software ingredient may not be exploitable in all products that contain that software ingredient. Exploitability would be determined in what code paths are taken via the product, and what other countermeasures may be in place in the overall product that obviate or mitigate the underlying software supply chain vulnerability.

There are no tools that can produce this enriched vulnerability data that includes vetting actual exploitability at scale. This ends up in the same resource crunch situation relying on skilled cybersecurity workers to make that final determination of risk and act upon it. […]

Much of the difficulty behind SBOM is that it does little to actually disclose potential vulnerabilities within a product’s software. We understand that by definition an SBOM does not include vulnerability information. That is why pushing for SBOMs is not immediately useful for increasing supply chain security. For example, if a product uses software A, and known vulnerability X exists in software A, it does not immediately follow that vulnerability X can be exploited in the product. The product may have other countermeasures and security processes in place to prevent vulnerability X (whether directly or indirectly), or it may just be that the coding paths in the product coupled with the use of software A does not provide a real security risk. While all of the data fields that NTIA identify are important, they do not reveal the whole picture.” - Luta Security (Katie Moussouris)

“Although there is value, we should recognize that use cases are limited to ONLY identifying component level vulnerabilities. Based on a SBOM one can only claim that the product/software has integrated a component with certain vulnerabilities. Whether this has any exposure beyond the product is determined by the design. SBOM is not usable for determining product level risk except for the product design owner. […] Note that multiple components can exist with the same name. And that components change their name over their life cycle, especially if the marketing department is involved.” - Philips

“An SBOM will not solve the issues of software supply chain security by itself. […] Fundamentally, SBOMs do not provide (and should not be relied on to be) the mechanism for software assurance, supply chain security, incident management, etc. […] To achieve the greatest degree of success, NTIA should […] recognize that SBOMs are not, in and of themselves, a solution to the cyber security challenge that the EO seeks to address.” - Red Hat

“SAFECode strongly believes that software developers should have an SBOM for any product they sell. […] Without an authoritative inventory of both open source and commercial third-party components (an SBOM), it is effectively impossible for a developer to assure the security of the software being delivered to customers.

As a practical matter, the use case for an SBOM relies on the actions of the software developer. The developer must monitor the quality and vulnerability status of third-party components, assess the impact of any vulnerabilities on the software in which they are embedded, and take appropriate action in response. […] Even customers who have access to the SBOM are unlikely to have the in-depth knowledge of the embedding software to determine what action is appropriate to remediate or mitigate the effects of a vulnerability in a component. […]

Given this dependence on the developer, it is important to recognize that the value of an SBOM to software customers is limited to giving them confidence that the developer actually maintains an SBOM and is capable of following a process for promptly mitigating vulnerabilities in components upon discovery. The primary consumer of the SBOM is the software developer. Thus, there is little or no demonstrated value to standardized SBOM formats such as those listed in the request for comment. […]

In some cases, product developers may embed proprietary components whose inclusion is subject to a confidentiality agreement between component provider and product developer. Under those circumstances, the developer may be contractually prevented from disclosing the full product SBOM or may only be able to provide an SBOM under a nondisclosure agreement with the customer. Any guidance issued under the Executive Order should recognize and accommodate those limitations.

Much of the text in the NTIA request for comments describes benefits from an SBOM that are highly speculative. It is important that no guidance or requirements about SBOM be issued under the Executive Order until there are clear and complete demonstrations at scale that the putative benefits are in fact realizable.” - SAFECode (Microsoft, Google, Dell, Adobe, Siemens, VMWare, etc.)

“While an SBOM can be valuable in helping engineers and other stakeholders gain an understanding of the software, it is only part of the information needed to determine the potential impact of any given vulnerability. Without clear understanding of how these software components interact, the true impact of specific vulnerabilities cannot be assessed. A cursory reading of the SBOM, its components and associated vulnerabilities might lead to misinterpretations and wrong conclusions if the connections between the SBOM components are misunderstood. It is important to understand that SBOMs will not be a silver bullet in helping with vulnerability management.” - Salesforce

“Chamber member opinion on an SBOM runs the gamut from strong support to robust skepticism. […] We believe that both the business community and NTIA want policymakers to ensure that an SBOM works for both businesses and agencies rather than see it become, unintentionally, an unproductive procurement and/or regulatory instrument. […]

Relying on this data can be fraught with challenges. Software components can be broken down into increasingly smaller components and levels of complexity. The value of a dependency relationship is debatable from a supply chain management standpoint. A dependency relationship, for instance, is usually meaningful (e.g., functional) to the author of the software but not to the consumer or the entity that receives an SBOM. […]

A missing element in the notice is the ability to pinpoint components that are “not vulnerable as used” components. One difficulty with an SBOM is that it tells part of a story regarding software assurance but not the entire story. This issue speaks to mapping to vulnerability databases, but it will only tell you that there is a known vulnerability (e.g., CVE) in a certain component. However, if a component is not vulnerable, then such information can be misleading at worst and incomplete at best. It could lead to customers erroneously pushing vendors to upgrade libraries that do not need upgrading. And vendors may produce patches that do not need to be produced, amounting to expending scarce resources on low-priority issues.” - US Chamber of Commerce