Framework: Software: HDR Globus Automate: A Distributed Research Automation Platform

框架：软件：HDR Globus Automate：分布式研究自动化平台

• 批准号: 1835890
• 负责人: Ian Foster
• 金额: $ 200万
• 依托单位: University of Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-11-01 至 2022-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1835890&HistoricalAwards=false
• 关键词: Framework; Software; HDR; Globus; Automate

Rapid increases in data volumes and velocities are overwhelming finite human capabilities. Continued progress in science and engineering demands that we automate a broad spectrum of currently manual research data manipulation tasks, from transfer and sharing to acquisition, publication, indexing, analysis, and inference. To address this need, which arises across essentially all scientific disciplines, this project will work with scientists in astronomy, engineering, geosciences, materials science, and neurosciences to develop and apply Globus Automate, a distributed research automation platform. Its purpose is to increase productivity and research quality across many science disciplines by allowing scientists to offload the management of a broad range of data acquisition, manipulation, and analysis tasks to a cloud-hosted distributed research automation platform. By thus enabling scientists to hand off responsibility for managing frequently performed tasks, such as acquiring, analyzing, and storing data, Globus Automate will increase the productivity of scientific instruments and the scientists that use them.This project will expand the capabilities and reach of the highly successful Globus research data management platform. Globus combines a professionally operated cloud-hosted management service with Globus Connect software deployed on more than 12,000 storage system endpoints, spanning most research universities, NSF-funded compute facilities, and NSF disciplines. Users employ Globus web interfaces and APIs to drive data movement, synchronization, and sharing tasks at and among endpoints. This ability to hand off responsibility for such tasks to cloud-hosted management logic has enabled substantial increases in data management efficiency, and spurred development of a wide range of innovative data management applications. Globus Automate will extend Globus capabilities to produce a full-featured distributed research automation platform that will enable the reliable, secure, and efficient automation of a wide range of research data management and manipulation activities. It will extend intuitive trigger-action programming models, suitable for non-programming users, to enable the specification and execution of a series of actions. It will provide for the detection of data events both at Globus storage system endpoints (e.g., creation or modification of new data files, extraction of new metadata) and at other sources (e.g., completion or failure of Globus transfer tasks); the propagation of such events to a cloud-hosted orchestration engine for reliable, efficient, and secure processing; and the invocation of remote actions on Globus endpoints and other resources. The project will leverage these basic event mechanisms to implement solutions to challenging science problems associated with partner science projects, and create a library of automation flows, both general-purpose (e.g., data publication and data replication) and domain-specific (e.g., feature detection in experimental data). These data event mechanisms will be made available on all storage systems relevant to research (Globus already supports most on-premises and cloud systems) and integrated with the Python language and JupyterLab environment that have become popular in science, so that researchers can define and share data automation behaviors as simple Python programs. A quantitative and qualitative research agenda will analyze the usability and adoption of both the platform and the research automation paradigm.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

数据量和速度的快速增长压倒了有限的人类能力。科学和工程的持续进步要求我们自动化目前手动研究数据操作任务的范围，从传输和共享到获取，出版，索引，分析和推理。为了满足几乎所有科学学科的需求，该项目将与天文学，工程学，地球科学，材料科学和神经科学的科学家合作，开发和应用Globus Automate，一个分布式研究自动化平台。其目的是通过允许科学家将广泛的数据采集，操作和分析任务的管理卸载到云托管的分布式研究自动化平台上，从而提高许多科学学科的生产力和研究质量。通过使科学家能够移交管理频繁执行的任务（例如获取、分析和存储数据）的责任，Globus Automate将提高科学仪器和使用它们的科学家的生产力。该项目将扩展非常成功的Globus研究数据管理平台的功能和范围。Globus将专业运营的云托管管理服务与部署在12，000多个存储系统端点上的Globus Connect软件相结合，这些端点涵盖大多数研究型大学、NSF资助的计算设施和NSF学科。用户使用Globus Web接口和API来驱动端点处和端点之间的数据移动、同步和共享任务。这种将此类任务的责任移交给云托管管理逻辑的能力，使数据管理效率大幅提高，并刺激了各种创新数据管理应用程序的开发。Globus Automate将扩展Globus的功能，以产生一个功能齐全的分布式研究自动化平台，该平台将实现广泛的研究数据管理和操作活动的可靠，安全和高效的自动化。它将扩展适用于非编程用户的直观的行为编程模型，以实现一系列行为的规范和执行。它将提供对Globus存储系统端点（例如，新数据文件的创建或修改，新元数据的提取）和在其它源（例如，Globus传输任务的完成或失败）;将这些事件传播到云托管的编排引擎以进行可靠、高效和安全的处理;以及在Globus端点和其他资源上调用远程动作。该项目将利用这些基本的事件机制来实施与合作伙伴科学项目相关的具有挑战性的科学问题的解决方案，并创建一个自动化流程库，包括通用（例如，数据发布和数据复制）和域特定（例如，实验数据中的特征检测）。这些数据事件机制将在与研究相关的所有存储系统上提供（Globus已经支持大多数内部部署和云系统），并与Python语言和在科学界已经流行的Python语言环境集成，以便研究人员可以将数据自动化行为定义和共享为简单的Python程序。定量和定性的研究议程将分析平台和研究自动化范式的可用性和采用情况。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Helping Users Debug Trigger-Action Programs

• DOI: 10.1145/3569506
• 发表时间: 2022-12
• 期刊: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
• 作者: Lefan Zhang;Cyrus Zhou;M. Littman;Blase Ur;Shan Lu

Visualizing Differences to Improve End-User Understanding of Trigger-Action Programs

• DOI: 10.1145/3334480.3382940
• 发表时间: 2020-04
• 期刊: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems
• 作者: Valerie Zhao;Lefan Zhang;Bo Wang;Shan Lu;Blase Ur

KondoCloud: Improving Information Management in Cloud Storage via Recommendations Based on File Similarity

• DOI: 10.1145/3472749.3474736
• 发表时间: 2021-10
• 期刊: The 34th Annual ACM Symposium on User Interface Software and Technology
• 作者: Will Brackenbury;A. Mcnutt;K. Chard;Aaron J. Elmore;Blase Ur

Files of a Feather Flock Together? Measuring and Modeling How Users Perceive File Similarity in Cloud Storage

• DOI: 10.1145/3404835.3462845
• 发表时间: 2021-07
• 期刊: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
• 作者: Will Brackenbury;Galen Harrison;K. Chard;Aaron J. Elmore;Blase Ur

Summarizing Sets of Related ML-Driven Recommendations for Improving File Management in Cloud Storage

• DOI: 10.1145/3526113.3545704
• 发表时间: 2022-10
• 期刊: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
• 作者: Will Brackenbury;K. Chard;Aaron J. Elmore;Blase Ur