CCRI: ENS: Machine Learning Democratization via a Linked, Annotated Repository of Datasets

机器学习（ML）是计算机科学中发展最快的研究领域之一，它正在推动医学、科学、工程、商业等众多新应用。在过去的30年里，机器学习稳步发展的基石是在测试平台ML数据集上定期和持续地对不同ML算法进行系统的经验比较。然而，随着机器学习研究论文、算法和数据集的数量持续快速增长，机器学习研究人员面临着信息过载，几乎不可能系统地跟踪最新的性能进步。反过来，这给研究人员带来了显著的低效率，减缓了新ML研究和应用的进展速度。该项目将在现有的加州-欧文大学（UCI）机器学习库的成功基础上解决这些问题，该库是一个知名且广泛使用的ML测试平台数据集的在线公共库，ML研究人员使用该数据集来评估和跟踪ML算法开发的进展。该项目将通过有关数据集的更多信息和复制结果的步骤来扩展存储库;两者都可以显着提高机器学习研究人员的生产力。该项目将通过在其设计和运营中利用大量的ML社区参与和推广来实现这一目标，并将为来自不同背景的学生提供如何参与和促进机器学习研究的培训。鉴于ML算法现在正被应用于气候科学、司法决策和个性化医疗等领域的预测问题，该项目在ML算法的系统评估方面的科学可重复性方面的进步具有潜在的深远的社会和科学效益。现有的UCI机器学习知识库直接影响成千上万的ML研究人员和学生，提供了一套标准和广泛引用的测试数据集，以支持研究和教育。该项目将涉及构建下一版本的存储库，该存储库将为ML数据集提供丰富的元数据，将数据集链接到研究论文，并以排行榜风格自动提取元数据和性能数据。新的存储库还将通过允许用户在测试平台数据集上轻松验证经验ML结果，为可重复的科学提供系统支持。该项目将导致两个方面的研究进展。第一个方面将是开发从科学文献中提取元数据信息的新方法和算法。第二个方面将来自ML研究人员进行实验工作的方式的改进。提供工具来支持更广泛、更系统和更可重复的ML算法评估，将导致ML进步更强大、更好地校准，并且在现实环境中使用时更有可能运行良好。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Machine learning (ML) is one of the fastest growing research areas within computer science and is enabling a multitude of new applications that span medicine, the sciences, engineering, business, and more. A cornerstone in the steady progress of machine learning over the past 30 years has been regular and continuous systematic empirical comparisons of different ML algorithms on testbed ML datasets. However, as the numbers of ML research papers, algorithms, and datasets all continue to grow rapidly, ML researchers are faced with information overload, making it near impossible to keep track of the latest performance advancements in a systematic manner. In turn, this creates significant inefficiencies for researchers, slowing down the pace of advances in new ML research and applications. This project will address these issues by building upon the success of the existing University of California - Irvine (UCI) Machine Learning Repository, a well-known and widely-used online public repository of ML testbed datasets that ML researchers use to evaluate and track progress in ML algorithm development. This project will extend the Repository with additional information about the datasets and steps to replicate the results; both can significantly amplify the productivity of ML researchers. The project will do so by harnessing significant ML community engagement and outreach in its design and operation, and will also provide training to students from different backgrounds on how to engage with, and contribute to, machine learning research. Given that ML algorithms are now being applied to prediction problems in areas as diverse as climate science, judicial decisions, and personalized medicine, the advances in scientific reproducibility from this project, in terms of systematic evaluation of ML algorithms, have potentially far-reaching societal and scientific benefits. The existing UCI Machine Learning Repository directly impacts tens of thousands of ML researchers and students, by providing a standard and widely-cited set of testbed datasets to support both research and education. This project will involve building the next version of the Repository that will provide rich metadata for ML datasets, linking datasets to research papers and automatically extracting metadata and performance data in leaderboard style. The new Repository will also provide systematic support for reproducible science by allowing users to readily validate empirical ML results on testbed datasets. This project will lead to research advances in two aspects. The first aspect will be the development of new methods and algorithms for information extraction of metadata from the scientific literature. The second aspect will result from improvements in the way ML researchers carry out their experimental work. Providing tools to support broader, more systematic, and more reproducible evaluations of ML algorithms, will lead to ML advances that are more robust, better calibrated, and more likely to operate well when used in real-world environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.