Framework: Software: HDR Globus Automate: A Distributed Research Automation Platform

数据量和速度的快速增长压倒了有限的人类能力。科学和工程的持续进步要求我们自动化目前广泛的手动研究数据处理任务，从传输和共享到获取、发布、索引、分析和推断。为了满足这种基本上跨越所有科学学科的需求，该项目将与天文学、工程学、地球科学、材料科学和神经科学的科学家合作，开发和应用Globus Automate，这是一个分布式研究自动化平台。其目的是通过允许科学家将范围广泛的数据采集、处理和分析任务的管理工作转移到云托管的分布式研究自动化平台，从而提高许多科学学科的工作效率和研究质量。Globus Automate通过这样使科学家能够移交管理经常执行的任务(如获取、分析和存储数据)的职责，将提高科学仪器和使用它们的科学家的生产率。该项目将扩大非常成功的Globus研究数据管理平台的能力和覆盖范围。Globus将专业运营的云托管管理服务与部署在12,000多个存储系统终端上的Globus Connect软件相结合，涵盖大多数研究型大学、NSF资助的计算设施和NSF学科。用户使用Globus Web界面和API来推动终端上和终端之间的数据移动、同步和共享任务。这种将此类任务的责任移交给云托管管理逻辑的能力大幅提高了数据管理效率，并刺激了各种创新型数据管理应用程序的开发。Globus Automate将扩展Globus的能力，以生产一个功能齐全的分布式研究自动化平台，使各种研究数据管理和操作活动能够实现可靠、安全和高效的自动化。它将扩展直观的触发器-动作编程模型，适用于非编程用户，以支持一系列动作的规范和执行。它将提供对Globus存储系统端点(例如，创建或修改新数据文件、提取新元数据)和其他来源(例如，Globus传输任务的完成或失败)的数据事件的检测；将此类事件传播到云托管的协调引擎以进行可靠、高效和安全的处理；以及调用对Globus端点和其他资源的远程操作。该项目将利用这些基本事件机制来实施与伙伴科学项目相关的挑战性科学问题的解决方案，并创建一个自动化流程库，既有通用的(例如，数据发布和数据复制)，也有特定领域的(例如，实验数据中的特征检测)。这些数据事件机制将在所有与研究相关的存储系统上可用(Globus已经支持大多数内部部署和云系统)，并与在科学中变得流行的Python语言和JupyterLab环境相集成，以便研究人员可以将数据自动化行为定义和共享为简单的Python程序。定量和定性的研究议程将分析平台和研究自动化范例的可用性和采用率。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Rapid increases in data volumes and velocities are overwhelming finite human capabilities. Continued progress in science and engineering demands that we automate a broad spectrum of currently manual research data manipulation tasks, from transfer and sharing to acquisition, publication, indexing, analysis, and inference. To address this need, which arises across essentially all scientific disciplines, this project will work with scientists in astronomy, engineering, geosciences, materials science, and neurosciences to develop and apply Globus Automate, a distributed research automation platform. Its purpose is to increase productivity and research quality across many science disciplines by allowing scientists to offload the management of a broad range of data acquisition, manipulation, and analysis tasks to a cloud-hosted distributed research automation platform. By thus enabling scientists to hand off responsibility for managing frequently performed tasks, such as acquiring, analyzing, and storing data, Globus Automate will increase the productivity of scientific instruments and the scientists that use them.This project will expand the capabilities and reach of the highly successful Globus research data management platform. Globus combines a professionally operated cloud-hosted management service with Globus Connect software deployed on more than 12,000 storage system endpoints, spanning most research universities, NSF-funded compute facilities, and NSF disciplines. Users employ Globus web interfaces and APIs to drive data movement, synchronization, and sharing tasks at and among endpoints. This ability to hand off responsibility for such tasks to cloud-hosted management logic has enabled substantial increases in data management efficiency, and spurred development of a wide range of innovative data management applications. Globus Automate will extend Globus capabilities to produce a full-featured distributed research automation platform that will enable the reliable, secure, and efficient automation of a wide range of research data management and manipulation activities. It will extend intuitive trigger-action programming models, suitable for non-programming users, to enable the specification and execution of a series of actions. It will provide for the detection of data events both at Globus storage system endpoints (e.g., creation or modification of new data files, extraction of new metadata) and at other sources (e.g., completion or failure of Globus transfer tasks); the propagation of such events to a cloud-hosted orchestration engine for reliable, efficient, and secure processing; and the invocation of remote actions on Globus endpoints and other resources. The project will leverage these basic event mechanisms to implement solutions to challenging science problems associated with partner science projects, and create a library of automation flows, both general-purpose (e.g., data publication and data replication) and domain-specific (e.g., feature detection in experimental data). These data event mechanisms will be made available on all storage systems relevant to research (Globus already supports most on-premises and cloud systems) and integrated with the Python language and JupyterLab environment that have become popular in science, so that researchers can define and share data automation behaviors as simple Python programs. A quantitative and qualitative research agenda will analyze the usability and adoption of both the platform and the research automation paradigm.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.