TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time
TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time
复制标题
DOI：
--
复制
发表时间：
2018-07
期刊：
影响因子：
--
通讯作者：
Feargus Pendlebury;Fabio Pierazzi;Roberto Jordaney;Johannes Kinder;L. Cavallaro
中科院分区：
文献类型：
--
作者：
Feargus Pendlebury;Fabio Pierazzi;Roberto Jordaney;Johannes Kinder;L. Cavallaro
Android 恶意软件分类问题已解决吗？公布的 F1 分数高达 0.99，似乎没有什么改进的空间。在本文中，我们认为，由于两个普遍存在的实验偏差来源，结果通常会被夸大：由不代表现实世界部署的训练和测试数据分布引起的“空间偏差”；训练集和测试集的时间分割不正确导致的“时间偏差”，导致不可能的配置。我们提出了一组实验设计的空间和时间限制，以消除这两种偏差来源。我们引入了一个新的指标，它总结了分类器在现实环境中的预期鲁棒性，并且我们提出了一种算法来调整其性能。最后，我们演示了这如何使我们能够评估时间衰减的缓解策略，例如主动学习。我们在 TESSERACT 中实施了我们的解决方案，TESSERACT 是一个开源评估框架，用于在现实环境中比较恶意软件分类器。我们使用 TESSERACT 评估了文献中的三个 Android 恶意软件分类器，涉及三年多的 129K 个应用程序的数据集。我们的评估证实了早期发布的结果存在偏差，同时也揭示了反直觉的性能，并表明适当的调整可以带来显着的改进。
Is Android malware classification a solved problem? Published F1 scores of up to 0.99 appear to leave very little room for improvement. In this paper, we argue that results are commonly inflated due to two pervasive sources of experimental bias: "spatial bias" caused by distributions of training and testing data that are not representative of a real-world deployment; and "temporal bias" caused by incorrect time splits of training and testing sets, leading to impossible configurations. We propose a set of space and time constraints for experiment design that eliminates both sources of bias. We introduce a new metric that summarizes the expected robustness of a classifier in a real-world setting, and we present an algorithm to tune its performance. Finally, we demonstrate how this allows us to evaluate mitigation strategies for time decay such as active learning. We have implemented our solutions in TESSERACT, an open source evaluation framework for comparing malware classifiers in a realistic setting. We used TESSERACT to evaluate three Android malware classifiers from the literature on a dataset of 129K applications spanning over three years. Our evaluation confirms that earlier published results are biased, while also revealing counter-intuitive performance and showing that appropriate tuning can lead to significant improvements.