Framework: Software: HDR Globus Automate: A Distributed Research Automation Platform
框架:软件:HDR Globus Automate:分布式研究自动化平台
基本信息
- 批准号:1835890
- 负责人:
- 金额:$ 200万
- 依托单位:
- 依托单位国家:美国
- 项目类别:Standard Grant
- 财政年份:2018
- 资助国家:美国
- 起止时间:2018-11-01 至 2022-10-31
- 项目状态:已结题
- 来源:
- 关键词:
项目摘要
Rapid increases in data volumes and velocities are overwhelming finite human capabilities. Continued progress in science and engineering demands that we automate a broad spectrum of currently manual research data manipulation tasks, from transfer and sharing to acquisition, publication, indexing, analysis, and inference. To address this need, which arises across essentially all scientific disciplines, this project will work with scientists in astronomy, engineering, geosciences, materials science, and neurosciences to develop and apply Globus Automate, a distributed research automation platform. Its purpose is to increase productivity and research quality across many science disciplines by allowing scientists to offload the management of a broad range of data acquisition, manipulation, and analysis tasks to a cloud-hosted distributed research automation platform. By thus enabling scientists to hand off responsibility for managing frequently performed tasks, such as acquiring, analyzing, and storing data, Globus Automate will increase the productivity of scientific instruments and the scientists that use them.This project will expand the capabilities and reach of the highly successful Globus research data management platform. Globus combines a professionally operated cloud-hosted management service with Globus Connect software deployed on more than 12,000 storage system endpoints, spanning most research universities, NSF-funded compute facilities, and NSF disciplines. Users employ Globus web interfaces and APIs to drive data movement, synchronization, and sharing tasks at and among endpoints. This ability to hand off responsibility for such tasks to cloud-hosted management logic has enabled substantial increases in data management efficiency, and spurred development of a wide range of innovative data management applications. Globus Automate will extend Globus capabilities to produce a full-featured distributed research automation platform that will enable the reliable, secure, and efficient automation of a wide range of research data management and manipulation activities. It will extend intuitive trigger-action programming models, suitable for non-programming users, to enable the specification and execution of a series of actions. It will provide for the detection of data events both at Globus storage system endpoints (e.g., creation or modification of new data files, extraction of new metadata) and at other sources (e.g., completion or failure of Globus transfer tasks); the propagation of such events to a cloud-hosted orchestration engine for reliable, efficient, and secure processing; and the invocation of remote actions on Globus endpoints and other resources. The project will leverage these basic event mechanisms to implement solutions to challenging science problems associated with partner science projects, and create a library of automation flows, both general-purpose (e.g., data publication and data replication) and domain-specific (e.g., feature detection in experimental data). These data event mechanisms will be made available on all storage systems relevant to research (Globus already supports most on-premises and cloud systems) and integrated with the Python language and JupyterLab environment that have become popular in science, so that researchers can define and share data automation behaviors as simple Python programs. A quantitative and qualitative research agenda will analyze the usability and adoption of both the platform and the research automation paradigm.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
数据量和速度的快速增长压倒了有限的人类能力。科学和工程的持续进步要求我们自动化目前手动研究数据操作任务的范围,从传输和共享到获取,出版,索引,分析和推理。为了满足几乎所有科学学科的需求,该项目将与天文学,工程学,地球科学,材料科学和神经科学的科学家合作,开发和应用Globus Automate,一个分布式研究自动化平台。其目的是通过允许科学家将广泛的数据采集,操作和分析任务的管理卸载到云托管的分布式研究自动化平台上,从而提高许多科学学科的生产力和研究质量。通过使科学家能够移交管理频繁执行的任务(例如获取、分析和存储数据)的责任,Globus Automate将提高科学仪器和使用它们的科学家的生产力。该项目将扩展非常成功的Globus研究数据管理平台的功能和范围。Globus将专业运营的云托管管理服务与部署在12,000多个存储系统端点上的Globus Connect软件相结合,这些端点涵盖大多数研究型大学、NSF资助的计算设施和NSF学科。用户使用Globus Web接口和API来驱动端点处和端点之间的数据移动、同步和共享任务。这种将此类任务的责任移交给云托管管理逻辑的能力,使数据管理效率大幅提高,并刺激了各种创新数据管理应用程序的开发。Globus Automate将扩展Globus的功能,以产生一个功能齐全的分布式研究自动化平台,该平台将实现广泛的研究数据管理和操作活动的可靠,安全和高效的自动化。它将扩展适用于非编程用户的直观的行为编程模型,以实现一系列行为的规范和执行。它将提供对Globus存储系统端点(例如,新数据文件的创建或修改,新元数据的提取)和在其它源(例如,Globus传输任务的完成或失败);将这些事件传播到云托管的编排引擎以进行可靠、高效和安全的处理;以及在Globus端点和其他资源上调用远程动作。该项目将利用这些基本的事件机制来实施与合作伙伴科学项目相关的具有挑战性的科学问题的解决方案,并创建一个自动化流程库,包括通用(例如,数据发布和数据复制)和域特定(例如,实验数据中的特征检测)。这些数据事件机制将在与研究相关的所有存储系统上提供(Globus已经支持大多数内部部署和云系统),并与Python语言和在科学界已经流行的Python语言环境集成,以便研究人员可以将数据自动化行为定义和共享为简单的Python程序。定量和定性的研究议程将分析平台和研究自动化范式的可用性和采用情况。该奖项反映了NSF的法定使命,并通过使用基金会的知识价值和更广泛的影响审查标准进行评估,被认为值得支持。
项目成果
期刊论文数量(13)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
Helping Users Debug Trigger-Action Programs
- DOI:10.1145/3569506
- 发表时间:2022-12
- 期刊:
- 影响因子:0
- 作者:Lefan Zhang;Cyrus Zhou;M. Littman;Blase Ur;Shan Lu
- 通讯作者:Lefan Zhang;Cyrus Zhou;M. Littman;Blase Ur;Shan Lu
Visualizing Differences to Improve End-User Understanding of Trigger-Action Programs
- DOI:10.1145/3334480.3382940
- 发表时间:2020-04
- 期刊:
- 影响因子:0
- 作者:Valerie Zhao;Lefan Zhang;Bo Wang;Shan Lu;Blase Ur
- 通讯作者:Valerie Zhao;Lefan Zhang;Bo Wang;Shan Lu;Blase Ur
KondoCloud: Improving Information Management in Cloud Storage via Recommendations Based on File Similarity
- DOI:10.1145/3472749.3474736
- 发表时间:2021-10
- 期刊:
- 影响因子:0
- 作者:Will Brackenbury;A. Mcnutt;K. Chard;Aaron J. Elmore;Blase Ur
- 通讯作者:Will Brackenbury;A. Mcnutt;K. Chard;Aaron J. Elmore;Blase Ur
Files of a Feather Flock Together? Measuring and Modeling How Users Perceive File Similarity in Cloud Storage
- DOI:10.1145/3404835.3462845
- 发表时间:2021-07
- 期刊:
- 影响因子:0
- 作者:Will Brackenbury;Galen Harrison;K. Chard;Aaron J. Elmore;Blase Ur
- 通讯作者:Will Brackenbury;Galen Harrison;K. Chard;Aaron J. Elmore;Blase Ur
Summarizing Sets of Related ML-Driven Recommendations for Improving File Management in Cloud Storage
- DOI:10.1145/3526113.3545704
- 发表时间:2022-10
- 期刊:
- 影响因子:0
- 作者:Will Brackenbury;K. Chard;Aaron J. Elmore;Blase Ur
- 通讯作者:Will Brackenbury;K. Chard;Aaron J. Elmore;Blase Ur
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
数据更新时间:{{ journalArticles.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ monograph.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ sciAawards.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ conferencePapers.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ patent.updateTime }}
Ian Foster其他文献
GreenFaaS: Maximizing Energy Efficiency of HPC Workloads with FaaS
GreenFaaS:利用 FaaS 最大限度提高 HPC 工作负载的能源效率
- DOI:
- 发表时间:
2024 - 期刊:
- 影响因子:0
- 作者:
Alok V. Kamatar;Valerie Hayot;Y. Babuji;André Bauer;Gourav Rattihalli;Ninad Hogade;D. Milojicic;Kyle Chard;Ian Foster - 通讯作者:
Ian Foster
DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies
DeepSpeed4Science 计划:通过复杂的人工智能系统技术实现大规模科学发现
- DOI:
- 发表时间:
2023 - 期刊:
- 影响因子:0
- 作者:
S. Song;Bonnie Kruft;Minjia Zhang;Conglong Li;Shiyang Chen;Chengming Zhang;Masahiro Tanaka;Xiaoxia Wu;Jeff Rasley;A. A. Awan;Connor Holmes;Martin Cai;Adam Ghanem;Zhongzhu Zhou;Yuxiong He;Christopher Bishop;Max Welling;Tie;Christian Bodnar;Johannes Brandsetter;W. Bruinsma;Chan Cao;Yuan Chen;Peggy Dai;P. Garvan;Liang He;E. Heider;Pipi Hu;Peiran Jin;Fusong Ju;Yatao Li;Chang Liu;Renqian Luo;Qilong Meng;Frank Noé;Tao Qin;Janwei Zhu;Bin Shao;Yu Shi;Wen;Gregor Simm;Megan Stanley;Lixin Sun;Yue Wang;Tong Wang;Zun Wang;Lijun Wu;Yingce Xia;Leo Xia;Shufang Xie;Shuxin Zheng;Jianwei Zhu;Pete Luferenko;Divya Kumar;Jonathan Weyn;Ruixiong Zhang;Sylwester Klocek;V. Vragov;Mohammed Alquraishi;Gustaf Ahdritz;C. Floristean;Cristina Negri;R. Kotamarthi;V. Vishwanath;Arvind Ramanathan;Sam Foreman;Kyle Hippe;T. Arcomano;R. Maulik;Max Zvyagin;Alexander Brace;Bin Zhang;Cindy Orozco Bohorquez;Austin R. Clyde;B. Kale;Danilo Perez;Heng Ma;Carla M. Mann;Michael Irvin;J. G. Pauloski;Logan Ward;Valerie Hayot;M. Emani;Zhen Xie;Diangen Lin;Maulik Shukla;Thomas Gibbs;Ian Foster;James J. Davis;M. Papka;Thomas Brettin;Prasanna Balaprakash;Gina Tourassi;John P. Gounley;Heidi Hanson;T. Potok;Massimiliano Lupo Pasini;Kate Evans;Dan Lu;D. Lunga;Junqi Yin;Sajal Dash;Feiyi Wang;M. Shankar;Isaac Lyngaas;Xiao Wang;Guojing Cong;Peifeng Zhang;Ming Fan;Siyan Liu;A. Hoisie;Shinjae Yoo;Yihui Ren;William Tang;K. Felker;Alexey Svyatkovskiy;Hang Liu;Ashwin Aji;Angela Dalton;Michael Schulte;Karl Schulz;Yuntian Deng;Weili Nie;Josh Romero;Christian Dallago;Arash Vahdat;Chaowei Xiao;Anima Anandkumar;R. Stevens - 通讯作者:
R. Stevens
An optical microscopy system for 3 D dynamic imagingRandy
用于 3D 动态成像的光学显微镜系统Randy
- DOI:
- 发表时间:
2007 - 期刊:
- 影响因子:0
- 作者:
R. Hudson;John N. Aarsvold;Chin;Jie Chen;Peter Davies;T. Disz;Ian Foster;Melvin Griem;Man K Kwong;B. Lin - 通讯作者:
B. Lin
Review of low-cost self-driving laboratories in chemistry and materials science: the “frugal twin” concept
化学与材料科学低成本自动驾驶实验室综述:“节俭双胞胎”概念
- DOI:
10.1039/d3dd00223c - 发表时间:
2024-05-15 - 期刊:
- 影响因子:5.600
- 作者:
Stanley Lo;Sterling G. Baird;Joshua Schrier;Ben Blaiszik;Nessa Carson;Ian Foster;Andrés Aguilar-Granda;Sergei V. Kalinin;Benji Maruyama;Maria Politi;Helen Tran;Taylor D. Sparks;Alán Aspuru-Guzik - 通讯作者:
Alán Aspuru-Guzik
Exploring Benchmarks for Self-Driving Labs using Color Matching
使用颜色匹配探索自动驾驶实验室的基准
- DOI:
- 发表时间:
2023 - 期刊:
- 影响因子:0
- 作者:
Tobias Ginsburg;Kyle Hippe;Ryan Lewis;Aileen Cleary;D. Ozgulbas;Rory Butler;Casey Stone;Abraham Stroka;Rafael Vescovi;Ian Foster - 通讯作者:
Ian Foster
Ian Foster的其他文献
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
{{ truncateString('Ian Foster', 18)}}的其他基金
Collaborative Research: NSF Workshop on Automated, Programmable and Self Driving Labs
合作研究:NSF 自动化、可编程和自动驾驶实验室研讨会
- 批准号:
2335910 - 财政年份:2023
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Frameworks: Garden: A FAIR Framework for Publishing and Applying AI Models for Translational Research in Science, Engineering, Education, and Industry
框架:Garden:用于发布和应用人工智能模型进行科学、工程、教育和工业转化研究的公平框架
- 批准号:
2209892 - 财政年份:2022
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: OAC Core: ScaDL: New Approaches to Scaling Deep Learning for Science Applications on Supercomputers
协作研究:OAC 核心:ScaDL:在超级计算机上扩展深度学习科学应用的新方法
- 批准号:
2107511 - 财政年份:2021
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
NSF Convergence Accelerator Track D: The Data Hypervisor: Orchestrating Data and Models
NSF 融合加速器轨道 D:数据管理程序:编排数据和模型
- 批准号:
2040718 - 财政年份:2020
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: Frameworks: funcX: A Function Execution Service for Portability and Performance
协作研究:框架:funcX:可移植性和性能的函数执行服务
- 批准号:
2004894 - 财政年份:2020
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Virtual Data Set Services Enabling New Science at NSF Facilities
虚拟数据集服务在 NSF 设施中实现新科学
- 批准号:
1841531 - 财政年份:2018
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
EAGER: Designing the OSN Software Platform
EAGER:设计 OSN 软件平台
- 批准号:
1836357 - 财政年份:2018
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
BD Spokes: SPOKE: MIDWEST: Collaborative: Integrative Materials Design (IMaD): Leverage, Innovate, and Disseminate
BD 辐条:辐条:中西部:协作:集成材料设计 (IMaD):利用、创新和传播
- 批准号:
1636950 - 财政年份:2017
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: CyberSEES:Type 2: Framework to Advance Climate, Economics, and Impact Investigations with Information Technology (FACE-IT)
合作研究:CyberSEES:类型 2:利用信息技术推进气候、经济和影响调查的框架 (FACE-IT)
- 批准号:
1331922 - 财政年份:2013
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: Managing Cloud Usage Allocation and Accounting for the NSF Community
协作研究:管理 NSF 社区的云使用分配和核算
- 批准号:
1250555 - 财政年份:2012
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
相似海外基金
Collaborative Research: Framework: Software: HDR: Reproducible Visual Analysis of Multivariate Networks with MultiNet
合作研究:框架:软件:HDR:使用 MultiNet 对多元网络进行可重复的视觉分析
- 批准号:
1835893 - 财政年份:2019
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: Framework: Software: HDR: Reproducible Visual Analysis of Multivariate Networks with MultiNet
合作研究:框架:软件:HDR:使用 MultiNet 对多元网络进行可重复的视觉分析
- 批准号:
1835904 - 财政年份:2019
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: Framework: Software: HDR: Building the Twenty-First Century Citizen Science Framework to Enable Scientific Discovery Across Disciplines
合作研究:框架:软件:HDR:构建二十一世纪公民科学框架以实现跨学科的科学发现
- 批准号:
1835272 - 财政年份:2019
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: Framework: Software: HDR: Building the Twenty-First Century Citizen Science Framework to Enable Scientific Discovery Across Disciplines
合作研究:框架:软件:HDR:构建二十一世纪公民科学框架以实现跨学科的科学发现
- 批准号:
1835530 - 财政年份:2019
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: Framework: Software: HDR: Building the Twenty-First Century Citizen Science Framework to Enable Scientific Discovery Across Disciplines
合作研究:框架:软件:HDR:构建二十一世纪公民科学框架以实现跨学科的科学发现
- 批准号:
1835632 - 财政年份:2019
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: Framework: Software: HDR: Building the Twenty-First Century Citizen Science Framework to Enable Scientific Discovery Across Disciplines
合作研究:框架:软件:HDR:构建二十一世纪公民科学框架以实现跨学科的科学发现
- 批准号:
1835574 - 财政年份:2019
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
NRT-HDR: Team training to develop new hardware and software applications for digital plant science across multiple scales
NRT-HDR:团队培训,为跨多个尺度的数字植物科学开发新的硬件和软件应用程序
- 批准号:
1922551 - 财政年份:2019
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: Framework: Software: HDR: Building the Twenty-First Century Citizen Science Framework to Enable Scientific Discovery Across Disciplines
合作研究:框架:软件:HDR:构建二十一世纪公民科学框架以实现跨学科的科学发现
- 批准号:
1835410 - 财政年份:2019
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Collaborative Research: Elements: Software: NSCI: HDR: Building An HPC/HTC Infrastructure For The Synthesis And Analysis Of Current And Future Cosmic Microwave Background Datasets
合作研究:要素:软件:NSCI:HDR:构建 HPC/HTC 基础设施以合成和分析当前和未来的宇宙微波背景数据集
- 批准号:
1835526 - 财政年份:2018
- 资助金额:
$ 200万 - 项目类别:
Standard Grant
Elements: Software: HDR: A knowledge base of deep time to facilitate automated workflows in studying the co-evolution of the geosphere and biosphere
要素:软件:HDR:促进研究地圈和生物圈共同进化的自动化工作流程的深度时间知识库
- 批准号:
1835717 - 财政年份:2018
- 资助金额:
$ 200万 - 项目类别:
Standard Grant