Disciplinary Improvements: AI Readiness, Reproducibility, and FAIR: Connecting Computing and Domain Communities Across the ML Lifecycle

该研究协调网络将在多学科社区中基于FAIR数据原则建立更好的实践，重点关注三个主题：机器学习中的FAIR、人工智能准备和可重复性。选择这些主题是为了解决地球物理和计算机科学研究人员的迫切需要。解决的一个关键问题是，机器学习模型通常使用默认的“最佳”参数（例如预训练模型）进行传播，但缺乏关于模型训练的文档、训练数据准备以及缺乏永久标识符。这阻碍了地球科学中机器学习的科学再现性，并且往往低估了模型输出的方差。该项目将以地球物理学界先前的成功为基础，利用现有的网络在这个新的研究协调网络中建立关系，从而创建一个网络的网络。与机器学习相关的专家和亲和小组将被召集起来，以了解新兴的最佳实践，利用哪些工具和资源，以及如何刺激实验，量化数据公平性与机器学习算法如何轻松有效地应用之间的关系，以及提高可重复性。地球科学数据存储库将更好地支持其用户使用机器学习方法准备、存储、访问和重用数据。RCN还将制定一份路线图，作为社区主导努力的指南，以在需要在人工智能研究中应用FAIR数据原则和开放科学的领域引起关注和资助。该项目将举办社区活动和工作组，收集所有三个主题的问题和实践。目标群体是地球物理数据存档提供商、研究人员自己创建的数据存档、机器学习科学家和从业者、高性能计算中心、围绕FAIR数据、大数据和人工智能的现有组织。该小组将协调社区活动，并编写一系列报告，包括回顾性和前瞻性指导方针。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This research coordination network will build better practices based on the FAIR data principles in multiple disciplinary communities, focusing on three themes: FAIR in machine learning, AI readiness, and reproducibility. These themes were chosen to address the urgent needs of researchers in the geophysical and computer sciences. A key problem addressed is that machine learning models are often disseminated with default “best” parameters (e.g. pre-trained models), but lack documentation on model training, data preparation for training, and lacking permanent identifiers. This hinders scientific reproducibility of machine learning in the geosciences and often underestimates the variance in model outputs. The project will build on prior successes in the geophysical community and utilize existing networks to build relationships in this new research coordination network, thereby creating a network of networks. Experts and affinity groups related to machine learning will be convened to understand emerging best practices, which tools and resources to leverage, and how to stimulate experimentation that quantifies the relationship between the FAIRness of data and how easily and efficiently machine learning algorithms can be applied, as well as advancing reproducibility. Geosciences data repositories will be better equipped to support their users in preparation, deposit, access, and reuse of data using machine learning methods. The RCN will also develop a roadmap that will serve as a guide for community-led efforts to spotlight attention and funding in areas where an application of FAIR data principles and open science in AI research is needed.The project will host community events and working groups to gather issues and practices for all three themes. The target community is geophysical data archive providers, data archives created by researchers themselves, machine learning scientists and practitioners, high performance computing centers, existing organizations revolving around FAIR data, big data, and AI. The team will coordinate community activities and create a series of reports including both retrospective and forward-looking guidelines.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.