A visualization framework for machine learning evaluation

在过去的25年里，机器学习，首先是数据挖掘，不断取得进步，现在这项技术已经应用于各行各业，如自动欺诈检测、推荐系统、设备监控等。然而，近年来，文献中报告的实际改进很少，特别是在分类领域。这可能意味着分类处于顶峰，不需要进一步研究；也可能意味着我们评估分类算法结果的方法过于简单，没有考虑我们系统的所有相关方面。鉴于其他领域的许多从业者不愿采用我们的技术，他们认为，这些技术没有经过充分的测试，值得一试，后者比前者更有可能。因此，这项拟议研究的目的是通过提议建立一个软件工具来解决分类器评估问题，该工具将允许对分类器进行可视评估，从而使用户能够以合理和人类可以理解的方式收集关于一个或几个系统的性能的大量相关信息。特别是，我们提出了一个新的框架，将分类器评估问题作为高维数据的可视化问题来处理。在这样做时，我们打算借用以前为该领域设计的工具(例如复杂的非线性预测)，并在这些工具的基础上加以扩展，以便能够进行复杂的评价程序，例如纳入统计保证和考虑纳入对阈值敏感的分类器。除了提供用于比较不同领域上的分类器的工具外，我们还打算开发工具，使我们能够将领域组织成等价的类，在这些等价类中，已知各种类型的分类器的行为是可预测的。我们还打算探索人工数据生成在机器学习评估中的作用，并提供工具来生成这样的领域。就像最近对ROC分析的研究一样，我们的工作旨在使评估过程更有意义，但与那项研究不同的是，它旨在易于使用和解释。

In the past 25 years, machine learning, first, and then, data mining, made constant progress to the point where the technology is now being used in all walks of life such as in automatic fraud detection, recommender systems, equipment monitoring and so on. In recent years, however, very few actual improvements were reported in the literature, particularly in the area of classification. This could either mean that classification is at its pinnacle and does not need to be investigated any further; or it could mean that our approach to evaluating the outcome of our classification algorithms is too simplistic and does not consider all the relevant aspects of our systems. Given the reticence of many practitioner in other fields to adopt our techniques which are not, they feel, tested to a level of adequacy that makes them worth their while, the latter is more likely than the former. The purpose of this proposed research is, thus, to address the issue of classifier evaluation by proposing to build a software tool that will allow classifiers to be evaluated visually so as to allow the user to gather a great deal of relevant information on the performance of one or several systems, in a way that is both sound and humanely understandable. In particular, we propose a new framework that approaches the problem of classifier evaluation as a problem of visualization of high-dimensional data. In so doing, we intend to borrow tools previously designed for that field (e.g., sophisticated nonlinear projections) and expand on these tools to allow for complex evaluation procedures such as the incorporation of statistical guarantees and the consideration of threshold-sensitive classifiers to be included. In addition to providing tools for comparing classifiers on various domains, we intend to develop tools that will allow us to organize domains into equivalence classes within which various types of classifiers are known to behave predictably. We also intend to explore the role of artificial data generation for machine learning evaluation and provide tools to generate such domains. Like recent research on ROC Analysis, our work is intended to make the evaluation process more meaningful, but unlike that research, it is intended to be easily usable and interpretable.