CCRI: ENS: Machine Learning Democratization via a Linked, Annotated Repository of Datasets

机器学习(ML)是计算机科学中发展最快的研究领域之一，它使许多新的应用成为可能，这些应用横跨医学、科学、工程、商业等等。在过去的30年里，机器学习稳步发展的一个基石是在测试床ML数据集上对不同的ML算法进行定期和连续的系统经验比较。然而，随着ML研究论文、算法和数据集的数量持续快速增长，ML研究人员面临着信息过载的问题，使得以系统的方式跟踪最新的性能进步几乎是不可能的。反过来，这对研究人员造成了严重的效率低下，减缓了新的ML研究和应用的进展速度。这个项目将通过建立现有的加州大学欧文分校(UCI)机器学习资源库的成功来解决这些问题，该资源库是一个著名的、广泛使用的ML试验台数据集的在线公共资源库，ML研究人员使用它来评估和跟踪ML算法开发的进展。该项目将使用有关数据集的附加信息和复制结果的步骤来扩展存储库；两者都可以显著提高ML研究人员的工作效率。该项目将通过在其设计和运营中利用ML社区的重大参与和扩展来做到这一点，并将为来自不同背景的学生提供如何参与和促进机器学习研究的培训。鉴于ML算法现在被应用于气候科学、司法判决和个性化医学等不同领域的预测问题，该项目在系统评估ML算法方面在科学重现性方面的进展，可能具有深远的社会和科学效益。现有的UCI机器学习资源库通过提供一套标准的和被广泛引用的试验台数据集来支持研究和教育，直接影响着数万名ML研究人员和学生。这个项目将包括建立下一个版本的资料库，它将为ML数据集提供丰富的元数据，将数据集链接到研究论文，并以排行榜的方式自动提取元数据和性能数据。新的资料库还将通过允许用户在试验台数据集上容易地验证经验最大似然结果，为可重复的科学提供系统支持。该项目将在两个方面带来研究进展。第一个方面是开发从科学文献中提取元数据信息的新方法和算法。第二个方面将来自ML研究人员开展实验工作的方式的改进。提供工具来支持更广泛、更系统和更可重复性的ML算法评估，将导致ML进步更强大、更好地校准，并且在实际环境中使用时更有可能运行良好。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning (ML) is one of the fastest growing research areas within computer science and is enabling a multitude of new applications that span medicine, the sciences, engineering, business, and more. A cornerstone in the steady progress of machine learning over the past 30 years has been regular and continuous systematic empirical comparisons of different ML algorithms on testbed ML datasets. However, as the numbers of ML research papers, algorithms, and datasets all continue to grow rapidly, ML researchers are faced with information overload, making it near impossible to keep track of the latest performance advancements in a systematic manner. In turn, this creates significant inefficiencies for researchers, slowing down the pace of advances in new ML research and applications. This project will address these issues by building upon the success of the existing University of California - Irvine (UCI) Machine Learning Repository, a well-known and widely-used online public repository of ML testbed datasets that ML researchers use to evaluate and track progress in ML algorithm development. This project will extend the Repository with additional information about the datasets and steps to replicate the results; both can significantly amplify the productivity of ML researchers. The project will do so by harnessing significant ML community engagement and outreach in its design and operation, and will also provide training to students from different backgrounds on how to engage with, and contribute to, machine learning research. Given that ML algorithms are now being applied to prediction problems in areas as diverse as climate science, judicial decisions, and personalized medicine, the advances in scientific reproducibility from this project, in terms of systematic evaluation of ML algorithms, have potentially far-reaching societal and scientific benefits. The existing UCI Machine Learning Repository directly impacts tens of thousands of ML researchers and students, by providing a standard and widely-cited set of testbed datasets to support both research and education. This project will involve building the next version of the Repository that will provide rich metadata for ML datasets, linking datasets to research papers and automatically extracting metadata and performance data in leaderboard style. The new Repository will also provide systematic support for reproducible science by allowing users to readily validate empirical ML results on testbed datasets. This project will lead to research advances in two aspects. The first aspect will be the development of new methods and algorithms for information extraction of metadata from the scientific literature. The second aspect will result from improvements in the way ML researchers carry out their experimental work. Providing tools to support broader, more systematic, and more reproducible evaluations of ML algorithms, will lead to ML advances that are more robust, better calibrated, and more likely to operate well when used in real-world environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.