Model Selection Competition for Very Large Datasets

国际和平协会将组织一次以模型选择为主题的机器学习比赛，并在主要会议上组织研讨会，顶级参与者将在研讨会上展示他们的结果。会议记录将作为特刊发表在《机器学习研究》杂志上。用于竞赛的平台将作为一项在线服务继续用于进一步的方法基准。大赛将对具有多个输入变量(特征)或多个条目(示例、模式)的行业数据集当前感兴趣的大量真实世界数据集进行分类和回归方法的基准测试。许多意味着10,000、100,000甚至一百万个特征和/或图案的数量级。例如，在视觉、文本处理、语音处理、生物信息学、天文学和组合化学(例如，用于高通量药物筛选)中遇到这样的问题。这些都带来了新的挑战，因为许多技术都是为较小的数据集开发的。人们早就认识到机器学习中基准的必要性，不是用玩具例子，而是用真实世界的数据。随着互联网的出现，数据交换变得非常容易，人们在发布结果时在网上发布数据已经很常见了。数据仓库使研究人员能够轻松地找到各种数据集以供选择。但是，由于各种原因，如果没有有组织的比赛，就很难进行比较。研究人员选择不同的数据集来测试他们的方法，有时只报告成功的数据。结果往往在统计学上并不显著。有时有一些基本的实验设计缺陷会使结果无效(例如，以一种微妙的方式在测试集上进行训练)。该项目将制定和实施合格的测试程序，以克服这些和相关的问题。

The PI will organize a machine learning competition on the theme of model selection and organize workshops at major conferences where top-ranking participants will present their results. The proceedings will be published as a special issue of the Journal of Machine Learning Research. The platform used for the competition will remain available for further method benchmarking as an on-line service. The competition will benchmark classification and regression methods on sizeable real-world data sets that are presently of interest to the industry datasets with either many input variables (features) or many entries (examples, patterns), or both. Many means of the order of 10,000, 100,000 or even one million features and/or patterns. Such problems are encountered, for instance, in vision, text processing, speech processing, bioinformatics, astronomy, and combinatorial chemistry (e.g. for high throughput drug screening). These pose new challenges because many techniques were developed for smaller size data sets. People have long identified the need for benchmarks in machine learning, not with toy examples but with real-world data. With the advent of the Internet, data exchange has become very easy and it is now very common that people publish their data on-line when they publish results. Data repositories allow researcher to find easily a variety of data sets from which to choose. But, without organized competitions, comparisons can hardly be made, for various reasons. Researchers choose different data sets on which they test their method, sometimes reporting only successes. The results are often not statistically significant. There are sometimes fundamental experimental design flaws that invalidate the results (e.g. training on the test set, in a subtle way). This project will develop and implement competent testing procedures to overcome these and related problems.