A visualization framework for machine learning evaluation

点击翻译按钮获取中文摘要

In the past 25 years, machine learning, first, and then, data mining, made constant progress to the point where the technology is now being used in all walks of life such as in automatic fraud detection, recommender systems, equipment monitoring and so on. In recent years, however, very few actual improvements were reported in the literature, particularly in the area of classification. This could either mean that classification is at its pinnacle and does not need to be investigated any further; or it could mean that our approach to evaluating the outcome of our classification algorithms is too simplistic and does not consider all the relevant aspects of our systems. Given the reticence of many practitioner in other fields to adopt our techniques which are not, they feel, tested to a level of adequacy that makes them worth their while, the latter is more likely than the former. The purpose of this proposed research is, thus, to address the issue of classifier evaluation by proposing to build a software tool that will allow classifiers to be evaluated visually so as to allow the user to gather a great deal of relevant information on the performance of one or several systems, in a way that is both sound and humanely understandable. In particular, we propose a new framework that approaches the problem of classifier evaluation as a problem of visualization of high-dimensional data. In so doing, we intend to borrow tools previously designed for that field (e.g., sophisticated nonlinear projections) and expand on these tools to allow for complex evaluation procedures such as the incorporation of statistical guarantees and the consideration of threshold-sensitive classifiers to be included. In addition to providing tools for comparing classifiers on various domains, we intend to develop tools that will allow us to organize domains into equivalence classes within which various types of classifiers are known to behave predictably. We also intend to explore the role of artificial data generation for machine learning evaluation and provide tools to generate such domains. Like recent research on ROC Analysis, our work is intended to make the evaluation process more meaningful, but unlike that research, it is intended to be easily usable and interpretable.