Understanding the Reproducibility of Crowd-reported Security Vulnerabilities
Understanding the Reproducibility of Crowd-reported Security Vulnerabilities
复制标题
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Dongliang Mu;A. Cuevas;Limin Yang;Hang Hu;Xinyu Xing;Bing Mao;G. Wang
中科院分区：
文献类型：
--
作者：
Dongliang Mu;A. Cuevas;Limin Yang;Hang Hu;Xinyu Xing;Bing Mao;G. Wang
今天的软件系统越来越依赖于“群体的力量”来识别新的安全漏洞。然而，人们还不清楚群众报告的漏洞有多可重复。在本文中，我们对各种现实世界的安全漏洞（共368个）进行了首次实证分析，目的是量化其可重复性。在严格控制的工作流程之后，我们组织了一个由安全分析师组成的专门小组进行复制实验。花费了3600个工时，我们获得了关于漏洞报告中缺失信息的普遍性和漏洞的低再现性的定量证据。我们发现，依靠流行的安全论坛的单一漏洞报告通常很难成功，因为信息不完整。通过广泛的众包信息收集，安全分析师可以提高复制成功率，但仍然面临着解决不可复制案例的关键挑战。为了进一步探索解决方案，我们调查了在软件安全领域拥有广泛专业知识的黑客、研究人员和工程师（N=43）。超越互联网规模的众包，我们发现，安全专业人员严重依赖手动调试和推测性猜测来推断丢失的信息。我们的研究结果表明，不仅有必要彻底改革安全论坛收集漏洞报告的方式，而且还需要自动化机制来收集报告中通常缺失的信息。
Today’s software systems are increasingly relying on the “power of the crowd” to identify new security vulnerabilities. And yet, it is not well understood how reproducible the crowd-reported vulnerabilities are. In this paper, we perform the first empirical analysis on a wide range of real-world security vulnerabilities (368 in total) with the goal of quantifying their reproducibility. Following a carefully controlled workflow, we organize a focused group of security analysts to carry out reproduction experiments. With 3600 man-hours spent, we obtain quantitative evidence on the prevalence of missing information in vulnerability reports and the low reproducibility of the vulnerabilities. We find that relying on a single vulnerability report from a popular security forum is generally difficult to succeed due to the incomplete information. By widely crowdsourcing the information gathering, security analysts could increase the reproduction success rate, but still face key challenges to trou-bleshoot the non-reproducible cases. To further explore solutions, we surveyed hackers, researchers, and engineers who have extensive domain expertise in software security (N=43). Going beyond Internet-scale crowd-sourcing, we find that, security professionals heavily rely on manual debugging and speculative guessing to infer the missed information. Our result suggests that there is not only a necessity to overhaul the way a security forum collects vulnerability reports, but also a need for automated mechanisms to collect information commonly missing in a report.