08-08, 18:00–18:45 (US/Pacific), Florentine A
In today's software development landscape, vulnerability scanners and SCA tools play a vital role in identifying potentially vulnerable software components and mitigating associated risks. However, their effectiveness remains questionable due to differences in implementation, coverage, and performance, as well as inherent blindspots that make them oblivious to critical vulnerabilities in real-world scenarios.
In this talk, we will present the results of a groundbreaking benchmark and root cause analysis research that evaluated leading commercial and open-source vulnerability scanners and SCA tools. We will showcase the main causes of scanner misidentifications, including blindspots created by common build and deployment practices, and thousands of hidden vulnerabilities we identified in real-world applications, many of which are known to be exploited in the wild.
Our findings expose a significant gap in the effectiveness of these tools and raise awareness about the need for objective evaluation criteria. Attendees will leave with a better understanding of the limitations of vulnerability scanners and SCA tools, as well as the importance of adopting more holistic approaches to software security.
- Introduction (5 minutes)
Explain the importance of visibility into what software components exist in your environment. We will discuss a few examples of recent vulnerabilities for which detection of the vulnerable component has proven to be especially challenging.
Explain how vulnerability scanners and SCA operate, and what their limitations are.
Specifically, we will highlight the fact that two separate processes have to take place for an SCA tool or vulnerability scanner to correctly report on existing vulnerabilities:
- Identification of existing components
- Mapping the identified components to vulnerabilities using the relevant security advisories.
Suboptimal performance in any of these stages will result in misidentifications whether false positive or false negative results.
So if you think about it, every scanner/SCA tool is also an SBOM tool. And the quality of its results is highly affected by the quality of the SBOM it is able to generate.
-
Evaluation method (5 minutes)
In this section, we will explain how the research was conducted and the results were measured. We will explain the concepts of precision, recall, and F1-score that will be used throughout the talk to measure the different scanner's performance. -
Present and discuss the results (10 minutes)
We will present a performance evaluation of 10 popular open-source and commercial scanners and delve into the huge variability in results across the different environments scanned (20 popular open-source containers and multiple base OS images from various cloud providers). we will explore the root causes we identified for the missidentifications discovered in the research (both false-positive and false negative results). Examples include: packages installed not via package managers, reliance on CPE (common platform enumeration), inaccurate/obsolete security advisory data, package and version identification errors, ignoring environment context, flagging kernel vulnerabilities in a container without taking into account the kernel version running on the host, and more.
- Hidden vulnerabilities (15 minutes)
In this section, we will deep dive into the concept of hidden vulnerabilities, which are known vulnerabilities that due to the inherent method of operation of vulnerability scanners and SCA tools remain invisible in a vulnerability scan.
We will show how wide this phenomenon is and discuss multiple common real-world scenarios, build and deployment practices that can unknowingly cause blindspots for security tools.
We will then provide examples of critical hidden vulnerabilities we identified in popular open-source applications and demonstrate how they can be exploited while remaining undetected in vulnerability scans.
Present the research outcomes: we have opened dozens of issues and notified the vendors of the different inaccuracies accounting for over 1000 unique CVEs either falsely identified or missed, some of the issues were already addressed.
-
Final Notes (5 minutes)
In this section, we will review the challenges presented in the talk and offer recommendations for both developers as well as security practitioners as to how to address them.
We will also call for vendors and maintainers to take these inherent gaps into account and implement logic that will be able to detect such vulnerabilities going forward. -
open the floor for questions/discussion (5 minutes)