A visualization framework for machine learning evaluation

在过去的25年里，机器学习，然后是数据挖掘，不断取得进展，该技术现在被用于各行各业，如自动欺诈检测，推荐系统，设备监控等。然而，近年来，在文献中很少有实际的改进，特别是在分类领域。这可能意味着分类处于顶峰，不需要进一步研究;或者它可能意味着我们评估分类算法结果的方法过于简单，没有考虑我们系统的所有相关方面。考虑到其他领域的许多从业者不愿采用我们的技术，他们认为，这些技术没有经过足够的测试，使他们值得一试，后者比前者更有可能。本研究的目的是，因此，提出建立一个软件工具，将允许分类器进行可视化评估，以便让用户收集大量的相关信息的性能上的一个或几个系统，在一种方式，是声音和人性化的理解，以解决分类器评价的问题。特别是，我们提出了一个新的框架，方法的分类器评估的问题作为一个问题的可视化高维数据。在这样做时，我们打算借用以前为该领域设计的工具（例如，复杂的非线性投影），并扩大这些工具，以允许复杂的评估程序，如纳入统计保证和考虑阈值敏感的分类器。除了提供工具比较分类器在不同的领域，我们打算开发工具，使我们能够组织域到等价类，其中各种类型的分类器是已知的行为可预测的。我们还打算探索人工数据生成在机器学习评估中的作用，并提供生成此类领域的工具。就像最近对ROC分析的研究一样，我们的工作旨在使评估过程更有意义，但与该研究不同的是，它旨在易于使用和解释。

In the past 25 years, machine learning, first, and then, data mining, made constant progress to the point where the technology is now being used in all walks of life such as in automatic fraud detection, recommender systems, equipment monitoring and so on. In recent years, however, very few actual improvements were reported in the literature, particularly in the area of classification. This could either mean that classification is at its pinnacle and does not need to be investigated any further; or it could mean that our approach to evaluating the outcome of our classification algorithms is too simplistic and does not consider all the relevant aspects of our systems. Given the reticence of many practitioner in other fields to adopt our techniques which are not, they feel, tested to a level of adequacy that makes them worth their while, the latter is more likely than the former. The purpose of this proposed research is, thus, to address the issue of classifier evaluation by proposing to build a software tool that will allow classifiers to be evaluated visually so as to allow the user to gather a great deal of relevant information on the performance of one or several systems, in a way that is both sound and humanely understandable. In particular, we propose a new framework that approaches the problem of classifier evaluation as a problem of visualization of high-dimensional data. In so doing, we intend to borrow tools previously designed for that field (e.g., sophisticated nonlinear projections) and expand on these tools to allow for complex evaluation procedures such as the incorporation of statistical guarantees and the consideration of threshold-sensitive classifiers to be included. In addition to providing tools for comparing classifiers on various domains, we intend to develop tools that will allow us to organize domains into equivalence classes within which various types of classifiers are known to behave predictably. We also intend to explore the role of artificial data generation for machine learning evaluation and provide tools to generate such domains. Like recent research on ROC Analysis, our work is intended to make the evaluation process more meaningful, but unlike that research, it is intended to be easily usable and interpretable.