Collaborative Research: Elements: TRAnsparency CErtified (TRACE): Trusting Computational Research Without Repeating It

自然科学和社会科学的研究团体越来越关注通过计算手段获得的结果的透明度和可重复性。在通过科学门户、数据门户和统计机构提供的研究数据产品的创建过程中，可以在同行评议期刊的政策和使用的处理管道中找到增加透明度的呼吁。这些社区认识到，当无法追踪其血统或验证其生产时，公布的结果和数据产品的完整性是不确定的。通过重复计算和比较结果来验证计算工件的透明度或可重复性是昂贵、耗时和困难的，如果研究产品依赖于受合法限制的资源（如使用敏感或专有数据），则可能是不可行的；流动的、暂时的或短暂的数据；大规模或专门的计算资源只提供给批准或授权的用户。TRACE项目通过一种被称为认证透明的方法来解决这个问题——一种由执行计算的系统签名的可靠计算记录。使用TRACE，系统所有者和操作人员可以验证产生结果或数据产品的计算工作流的原始执行。通过使用支持trace的系统，研究人员产生透明的计算工件，不再需要验证，从而减轻了期刊编辑和审稿人寻求确保计算结果的可重复性和透明度的负担。TRACE提供了一种创新和有效的方法来确保使用计算方法的研究的透明度，与美国国家科学院概述的愿景一致，并使基于透明和可信赖的科学的循证决策成为可能。TRACE项目的中心目标是开发、验证和实现经过认证的透明技术模型。这包括一组基础设施元素，系统所有者可以使用这些元素来(1)声明其平台支持的计算透明度的维度；(2)证明在平台上执行了特定的计算工作流；(3)将工件、它们的执行记录、关于它们内容的技术元数据捆绑在一起，并证明它们可以传播。项目的第一阶段侧重于概念模型和技术规范的开发，这些模型和技术规范可用于认证系统的描述，称为透明度认证系统（TRACE系统），以及工件的聚合及其执行记录，称为透明度认证研究对象（TROs）。第二阶段的重点是实现TRACE模型和方法的可重用软件组件的开发。为了演示经过认证的透明度，使用该工具包来启用trace现有平台，包括Whole Tale、SKOPE和SLURM工作负载管理器。这些启用trace的系统生成经过认证的tro，这些tro是可信的，不需要重复或重新执行来验证结果是否如声明的那样获得。该奖项由先进网络基础设施办公室颁发，并由社会、行为和经济科学局的社会和经济科学司联合支持；由计算机和信息科学与工程理事会的信息和智能系统司负责。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Research communities across the natural and social sciences are increasingly concerned about the transparency and reproducibility of results obtained by computational means. Calls for increased transparency can be found in the policies of peer-reviewed journals and processing pipelines employed in the creation of research data products made available through science gateways, data portals, and statistical agencies. These communities recognize that the integrity of published results and data products is uncertain when it is not possible to trace their lineage or validate their production. Verifying the transparency or reproducibility of computational artifacts—by repeating computations and comparing results—is expensive, time-consuming, and difficult, and may be infeasible if the research products rely on resources that are subject to legitimate restrictions such as the use of sensitive or proprietary data; streaming, transient, or ephemeral data; and large-scale or specialized computational resources available only to approved or authorized users. The TRACE project is addressing this problem through an approach called certified transparency - a trustworthy record of computations signed by the systems within which they were performed. Using TRACE, system owners and operators certify the original execution of a computational workflow that produces findings or data products. By using a TRACE-enabled system, researchers produce transparent computational artifacts that no longer require verification, reducing burden on journal editors and reviewers seeking to ensure reproducibility and transparency of computational results. TRACE presents an innovative and efficient approach to ensuring the transparency of research that uses computational methods, is consistent with the vision outlined by the National Academies, and enables evidence-based policymaking based on transparent and trustworthy science.The central goal of the TRACE project is the development, validation, and implementation of a technical model of certified transparency. This includes a set of infrastructure elements that can be employed by system owners to (1) declare the dimensions of computational transparency supported by their platforms; (2) certify that a specific computational workflow was executed on the platform; and (3) bundle artifacts, records of their execution, technical metadata about their contents, and certify them for dissemination. The first phase of the project focuses on the development of a conceptual model and technical specification that can be used to certify the description of a system, termed a Transparency-Certified System (TRACE system), and the aggregation of artifacts along with records of their execution, termed Transparency-Certified Research Objects (TROs). The second phase focuses on the development of reusable software components implementing the TRACE model and approach. To demonstrate certified transparency, the toolkit is used to TRACE-enable existing platforms including Whole Tale, SKOPE, and the SLURM workload manager. These TRACE-enabled systems produce certified TROs that can be trusted and do not need to be repeated or re-executed to verify that results were obtained as claimed.This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Division of Social and Economic Sciences within the Directorate for Social, Behavioral and Economic Sciences; and by the Division of Information and Intelligent Systems within the Directorate for Computer and Information Science and Engineering.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.