Collaborative Research: Frameworks: funcX: A Function Execution Service for Portability and Performance

协作研究：框架：funcX：可移植性和性能的函数执行服务

• 批准号: 2004932
• 负责人: Daniel Katz
• 金额: $ 48.12万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-05-01 至 2025-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2004932&HistoricalAwards=false
• 关键词: Collaborative; Research; Frameworks; funcX; Function

The funcX project is developing, deploying, and operating a new distributed computing cyberinfrastructure platform to enable researchers to build applications from programming functions that execute on different computing resources, from laptops to supercomputers. This cloud-hosted service democratizes access to advanced computing by providing intuitive interfaces for both registering remote computers as function executors and executing functions on these computers reliably, securely, and with high performance. Researchers can thus decompose monolithic applications into collections of reusable lightweight functions that can be run wherever makes the most sense, for example where data reside or where excess capacity is available. By simplifying access to specialized and high performance cyberinfrastructure and decreasing the time to discovery, the project serves the national interest, as stated in NSF's mission, by promoting the progress of science. A total of 33 diverse science, cyberinfrastructure, and software institute partners working with cutting-edge science applications and research cyberinfrastructure will directly benefit from the funcX platform.This project develops funcX, a scalable and high-performance federated platform for managing the remote execution of (often short-duration) functions across diverse cyberinfrastructure systems, from edge accelerators to clusters, supercomputers, and clouds. funcX allows developers to decompose applications into collections of functions that can each be executed in the best location, in terms of cost, execution time, data movement costs, and/or energy consumption. It thus integrates the extreme convenience of the function as a service (FaaS) model, developed in industry for specific industry applications, with support for the specialized needs of scientific research. funcX addresses important barriers to these new uses of research cyberinfrastructure systems, by enabling the intuitive, flexible, and scalable execution of functions without regard to physical location, scheduler architecture, virtualization technology, administrative domain, or data location. Flexible open-source funcX agent software makes it easy to expose arbitrary computing systems as funcX computing platforms, thereby transforming existing cyberinfrastructure systems into high-performance function serving environments (endpoints). The cloud-hosted funcX service provides a REST interface for registering functions, discovering available endpoints, and managing the execution of functions on endpoints, all via a universal trust fabric and standard web authentication and authorization mechanisms. It dynamically creates and deploys containers that incorporate function dependencies and provide a secure and isolated environment for safe function execution. The project engages a diverse set of 11 science partners, 18 research computing and cyberinfrastructure projects, and 4 NSF Software Institutes, each supporting many NSF-funded researchers, to provide use cases for funcX, shape its design, and evaluate its implementation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The funcX project is developing, deploying, and operating a new distributed computing cyberinfrastructure platform to enable researchers to build applications from programming functions that execute on different computing resources, from laptops to supercomputers. This cloud-hosted service democratizes access to advanced computing by providing intuitive interfaces for both registering remote computers as function executors and executing functions on these computers reliably, securely, and with high performance. Researchers can thus decompose monolithic applications into collections of reusable lightweight functions that can be run wherever makes the most sense, for example where data reside or where excess capacity is available. By simplifying access to specialized and high performance cyberinfrastructure and decreasing the time to discovery, the project serves the national interest, as stated in NSF's mission, by promoting the progress of science. A total of 33 diverse science, cyberinfrastructure, and software institute partners working with cutting-edge science applications and research cyberinfrastructure will directly benefit from the funcX platform.This project develops funcX, a scalable and high-performance federated platform for managing the remote execution of (often short-duration) functions across diverse cyberinfrastructure systems, from edge accelerators to clusters, supercomputers, and clouds. funcX allows developers to decompose applications into collections of functions that can each be executed in the best location, in terms of cost, execution time, data movement costs, and/or energy consumption. It thus integrates the extreme convenience of the function as a service (FaaS) model, developed in industry for specific industry applications, with support for the specialized needs of scientific research. funcX addresses important barriers to these new uses of research cyberinfrastructure systems, by enabling the intuitive, flexible, and scalable execution of functions without regard to physical location, scheduler architecture, virtualization technology, administrative domain, or data location. Flexible open-source funcX agent software makes it easy to expose arbitrary computing systems as funcX computing platforms, thereby transforming existing cyberinfrastructure systems into high-performance function serving environments (endpoints). The cloud-hosted funcX service provides a REST interface for registering functions, discovering available endpoints, and managing the execution of functions on endpoints, all via a universal trust fabric and standard web authentication and authorization mechanisms. It dynamically creates and deploys containers that incorporate function dependencies and provide a secure and isolated environment for safe function execution. The project engages a diverse set of 11 science partners, 18 research computing and cyberinfrastructure projects, and 4 NSF Software Institutes, each supporting many NSF-funded researchers, to provide use cases for funcX, shape its design, and evaluate its implementation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

项目成果

DLHub: Simplifying publication, discovery, and use of machine learning models in science

• DOI: 10.1016/j.jpdc.2020.08.006
• 发表时间: 2021-01-01
• 期刊: JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING
• 影响因子: 3.8
• 作者: Li, Zhuozhao;Chard, Ryan;Foster, Ian
• 通讯作者: Foster, Ian

Enhancing Automated FaaS with Cost-aware Provisioning of Cloud Resources

通过具有成本意识的云资源配置来增强自动化 FaaS

• DOI: 10.1109/escience51609.2021.00053
• 发表时间: 2021
• 期刊: 2021 IEEE 17th International Conference on eScience (eScience
• 作者: Baughman, Matt;Foster, Ian;Chard, Kyle
• 通讯作者: Chard, Kyle

? unc X: Federated Function as a Service for Science

？

• DOI: 10.1109/tpds.2022.3208767
• 发表时间: 2022
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Li, Zhuozhao;Chard, Ryan;Babuji, Yadu;Galewsky, Ben;Skluzacek, Tyler J.;Nagaitsev, Kirill;Woodard, Anna;Blaiszik, Ben;Bryan, Josh;Katz, Daniel S.
• 通讯作者: Katz, Daniel S.

A Serverless Framework for Distributed Bulk Metadata Extraction

用于分布式批量元数据提取的无服务器框架

• DOI: 10.1145/3431379.3460636
• 发表时间: 2021
• 期刊: Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing (HPDC
• 作者: Skluzacek, Tyler J.;Wong, Ryan;Li, Zhuozhao;Chard, Ryan;Chard, Kyle;Foster, Ian
• 通讯作者: Foster, Ian

An Empirical Study of Package Dependencies and Lifetimes in Binder Python Containers

Binder Python 容器中包依赖关系和生命周期的实证研究

• DOI: 10.1109/escience51609.2021.00032
• 发表时间: 2021
• 期刊: 2021 IEEE 17th International Conference on eScience (eScience
• 作者: Shaffer, Tim;Chard, Kyle;Thain, Douglas
• 通讯作者: Thain, Douglas