Detecting Training Abuses in Neural Nets

许多军事系统执行分类任务。例如，系统可能需要区分盟军坦克和敌方坦克（机器学习中的经典问题）。现代机器学习方法正在军事领域和更广泛的范围内用于分类问题。神经网络是一种技术，粗略地说，它以类似于人类大脑工作方式的方式做出决策，发挥着特别突出的作用。大多数工作都是在假设一切都是良性的基础上进行的。但是想象一下，如果一个敌人想让你的分类器工作得很好，除非有一个非常具体的分类任务。例如，一辆具有特定外观的敌方坦克可以被归类为盟军坦克，这会产生非常重大的后果。敌人能策划这样的行为吗？在某些情况下，是的！这取决于分类系统是如何建立的，以及由谁建立的。这类系统的建设往往以某种方式外包，例如，因为采购者缺乏制作有效系统的计算能力，或使用公共生成的组件。我们经常将隐藏的恶意功能称为“陷阱门”，这些功能可以在方便的时候调用。暗门通常很难被发现。想象一下，一个系统可以完美地分类您提供的数千个坦克示例。这似乎是一个非常好的系统。但该系统可能已经训练，使敌人的坦克与“666”画在其一侧是错误的分类。如果你不知道这个特定的条件，你就没有理由生成一个测试示例来发现它。神经网络在呈现它们如何做出决策方面也是出了名的不透明，这使得这种陷阱检测特别困难。我们可能会合理地问，我们是否能探测到这些陷阱门，或者探测得有多好。可以在不同层次上寻求理解。因此，确定系统中是否具有活板门（是/否）是比寻找特定活板门条件（上文指示的“666”）更简单且不那么雄心勃勃的任务。虽然在文献中有相当数量的活板门，通常解决种植或检测活板门的问题，但似乎很少关注它们的特征。很明显，任何探测技术在某些活板门上都可能比在其他活板门上更成功。然而，这就提出了一个问题，即如何描述那些技术效果良好的地方和那些技术效果不太好的地方。一个严格的检测方法，这个项目的主要目标，需要一个细致入微的了解陷阱。特别是，活板门的表征及其特性的测量，例如活板门实例偏离正常实例的程度，是必不可少的。如果现在考虑活板门生成，活板门的特征化允许我们希望插入的活板门具有的属性的更精细的规范。这有两个目的：首先，它促进了实际操作目的的更细致的世代能力，即对于想要从在真实的世界中种植活板门中受益的人;其次，它允许研究人员（最初是我们自己！）以生成用于检测技术的严格评估的陷门集合。我们可以用某种方式定义“覆盖”陷门空间的含义，就像我们在一般测试中覆盖输入或其他空间一样。由于活板门没有现存的可行的特征，显然也没有现存的世代能力。

Many military systems carry out classification tasks. For example, a system might be required to distinguish between an allied tank and an enemy tank (a classic problem in machine learning). Modern machine learning approaches are being brought to bear on classification problems within the military domain and more widely. Neural networks, a technology that, loosely speaking, makes decisions in a manner analogous to the way the human brain works, play a particularly prominent role. Most work proceeds on the assumption that all is benign. But imagine if an enemy wanted to cause your classifier to work well, except when presented with a very specific classification task. For example, an enemy tank with a particular appearance could be classified as an allied one, with very significant consequences. Can an enemy engineer such behaviour? In certain circumstances, yes! It depends on how and by whom the classifier system was built. The building of such systems is often outsourced in some way, e.g. because the procurer lacks the computational capability to craft an effective system or by the use of publicly generated components. We often refer to hidden malicious functionality that can be invoked when convenient as a 'trapdoor'. Trapdoors are often very difficult to detect. Imagine a system that classified perfectly thousands of tank examples provided by you. It seems like this is a very good system. But the system may have been trained so that an enemy tank with "666" painted on its side is misclassified. If you don't know this specific condition you would have little reason to generate a test example to discover it. Neural networks are also notoriously opaque in rendering apparent how they make decisions and this makes this sort of trapdoor detection particularly hard. We might reasonably ask whether or how well we can detect such trapdoors. There are various levels at which understanding may be sought. Thus, determining whether a system has a trapdoor in it (yes/no) is a simpler and less ambitious task than seeking the specific trapdoor condition (the "666' indicated above). Though there is a fair amount on trapdoors in the literature, typically addressing issues of planting or detecting trapdoors, there appears to be little concerned with characterising them. It would seem clear that any detection technique is likely to be more successful on some trapdoors than on others. This raises the question, however, as to how to describe those where the technique works well and those where it performs less well. A rigorous approach to detection, the primary goal of this project, requires a nuanced understanding of trapdoors. In particular, a characterisation of trapdoors together with measurements of their properties, e.g. how much a trapdoor example deviates from a normal example, is essential. If trapdoor generation is now considered, the characterisation of trapdoors allows more refined specification of properties we would like an inserted trapdoor to have. This serves two purposes: firstly, it facilitates a more nuanced generational capability for practical operational purposes, i.e. for someone who wants to benefit from planting a trapdoor in the real world; and secondly, it allows researchers (initially ourselves!) to generate sets of trapdoors for rigorous evaluation of detection techniques. We can define what it means to 'cover' the trapdoor space in some way, much as we cover input or other space in general testing. Since there is no extant workable characterisation of trapdoors there is also clearly no extant generational capability.