A visualization framework for machine learning evaluation

在过去的25年里，首先是机器学习，然后是数据挖掘，不断取得进步，现在该技术被应用于各行各业，如自动欺诈检测、推荐系统、设备监控等。然而，近年来，文献中很少报道实际的改进，特别是在分类领域。这可能意味着分类正处于顶峰，不需要进一步研究；或者这可能意味着我们评估分类算法结果的方法过于简单，没有考虑到我们系统的所有相关方面。鉴于其他领域的许多从业者对采用我们的技术持保留态度，因为他们认为，我们的技术没有经过足够的测试，因此，后者比前者更有可能被采用。因此，本研究的目的是通过构建一个软件工具来解决分类器评估的问题，该软件工具可以直观地评估分类器，从而使用户能够以一种既合理又人性化的方式收集大量有关一个或多个系统性能的相关信息。特别地，我们提出了一个新的框架，将分类器评估问题作为高维数据的可视化问题来处理。在这样做的过程中，我们打算借用以前为该领域设计的工具（例如，复杂的非线性预测），并对这些工具进行扩展，以允许复杂的评估程序，例如合并统计保证和考虑阈值敏感分类器。除了提供比较不同领域分类器的工具外，我们还打算开发一些工具，使我们能够将领域组织成等价类，在等价类中，各种类型的分类器的行为都是可预测的。我们还打算探索人工数据生成在机器学习评估中的作用，并提供工具来生成这样的领域。就像最近对ROC分析的研究一样，我们的工作旨在使评估过程更有意义，但与该研究不同的是，它旨在易于使用和解释。

In the past 25 years, machine learning, first, and then, data mining, made constant progress to the point where the technology is now being used in all walks of life such as in automatic fraud detection, recommender systems, equipment monitoring and so on. In recent years, however, very few actual improvements were reported in the literature, particularly in the area of classification. This could either mean that classification is at its pinnacle and does not need to be investigated any further; or it could mean that our approach to evaluating the outcome of our classification algorithms is too simplistic and does not consider all the relevant aspects of our systems. Given the reticence of many practitioner in other fields to adopt our techniques which are not, they feel, tested to a level of adequacy that makes them worth their while, the latter is more likely than the former. The purpose of this proposed research is, thus, to address the issue of classifier evaluation by proposing to build a software tool that will allow classifiers to be evaluated visually so as to allow the user to gather a great deal of relevant information on the performance of one or several systems, in a way that is both sound and humanely understandable. In particular, we propose a new framework that approaches the problem of classifier evaluation as a problem of visualization of high-dimensional data. In so doing, we intend to borrow tools previously designed for that field (e.g., sophisticated nonlinear projections) and expand on these tools to allow for complex evaluation procedures such as the incorporation of statistical guarantees and the consideration of threshold-sensitive classifiers to be included. In addition to providing tools for comparing classifiers on various domains, we intend to develop tools that will allow us to organize domains into equivalence classes within which various types of classifiers are known to behave predictably. We also intend to explore the role of artificial data generation for machine learning evaluation and provide tools to generate such domains. Like recent research on ROC Analysis, our work is intended to make the evaluation process more meaningful, but unlike that research, it is intended to be easily usable and interpretable.