Collaborative Research: Frameworks: funcX: A Function Execution Service for Portability and Performance

协作研究：框架：funcX：可移植性和性能的函数执行服务

• 批准号: 2004932
• 负责人: Daniel Katz
• 金额: $ 48.12万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-05-01 至 2025-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2004932&HistoricalAwards=false
• 关键词: Collaborative; Research; Frameworks; funcX; Function

The funcX project is developing, deploying, and operating a new distributed computing cyberinfrastructure platform to enable researchers to build applications from programming functions that execute on different computing resources, from laptops to supercomputers. This cloud-hosted service democratizes access to advanced computing by providing intuitive interfaces for both registering remote computers as function executors and executing functions on these computers reliably, securely, and with high performance. Researchers can thus decompose monolithic applications into collections of reusable lightweight functions that can be run wherever makes the most sense, for example where data reside or where excess capacity is available. By simplifying access to specialized and high performance cyberinfrastructure and decreasing the time to discovery, the project serves the national interest, as stated in NSF's mission, by promoting the progress of science. A total of 33 diverse science, cyberinfrastructure, and software institute partners working with cutting-edge science applications and research cyberinfrastructure will directly benefit from the funcX platform.This project develops funcX, a scalable and high-performance federated platform for managing the remote execution of (often short-duration) functions across diverse cyberinfrastructure systems, from edge accelerators to clusters, supercomputers, and clouds. funcX allows developers to decompose applications into collections of functions that can each be executed in the best location, in terms of cost, execution time, data movement costs, and/or energy consumption. It thus integrates the extreme convenience of the function as a service (FaaS) model, developed in industry for specific industry applications, with support for the specialized needs of scientific research. funcX addresses important barriers to these new uses of research cyberinfrastructure systems, by enabling the intuitive, flexible, and scalable execution of functions without regard to physical location, scheduler architecture, virtualization technology, administrative domain, or data location. Flexible open-source funcX agent software makes it easy to expose arbitrary computing systems as funcX computing platforms, thereby transforming existing cyberinfrastructure systems into high-performance function serving environments (endpoints). The cloud-hosted funcX service provides a REST interface for registering functions, discovering available endpoints, and managing the execution of functions on endpoints, all via a universal trust fabric and standard web authentication and authorization mechanisms. It dynamically creates and deploys containers that incorporate function dependencies and provide a secure and isolated environment for safe function execution. The project engages a diverse set of 11 science partners, 18 research computing and cyberinfrastructure projects, and 4 NSF Software Institutes, each supporting many NSF-funded researchers, to provide use cases for funcX, shape its design, and evaluate its implementation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

FuncX项目正在开发、部署和运营一个新的分布式计算网络基础设施平台，使研究人员能够通过编程功能构建在从笔记本电脑到超级计算机等不同计算资源上执行的应用程序。这项云托管服务通过为将远程计算机注册为功能执行者以及在这些计算机上可靠、安全和高性能地执行功能提供直观的界面，使对高级计算的访问变得大众化。因此，研究人员可以将单片应用程序分解为可重复使用的轻量级函数集合，这些函数可以在任何最有意义的地方运行，例如数据驻留的地方或过剩容量可用的地方。通过简化对专业和高性能网络基础设施的访问，减少发现时间，该项目通过促进科学进步来服务于国家利益，正如NSF的使命所述。共有33家致力于尖端科学应用和研究网络基础设施的不同科学、网络基础设施和软件研究所合作伙伴将直接受益于FuncX平台。该项目开发了一个可扩展的高性能联合平台，用于管理从边缘加速器到集群、超级计算机和云等不同网络基础设施系统的远程执行(通常是短期的)功能。FuncX允许开发人员将应用程序分解为功能集合，每个功能集合都可以在成本、执行时间、数据移动成本和/或能源消耗方面在最佳位置执行。因此，它将在工业中为特定行业应用开发的功能即服务(FAAS)模型的极大便利性与支持科学研究的专门需求结合在一起。通过实现直观、灵活且可扩展的功能执行，而无需考虑物理位置、调度程序架构、虚拟化技术、管理域或数据位置，FuncX解决了这些研究网络基础设施系统新用途的重要障碍。灵活的开源函数X代理软件使得将任意计算系统暴露为函数X计算平台变得容易，从而将现有的网络基础设施系统转变为高性能的功能服务环境(端点)。云托管的FuncX服务提供REST接口，用于注册功能、发现可用端点以及管理端点上功能的执行，所有这些都通过通用信任结构和标准Web身份验证和授权机制实现。它动态地创建和部署包含函数依赖项的容器，并为安全的函数执行提供安全和隔离的环境。该项目涉及11个科学合作伙伴、18个研究计算和网络基础设施项目以及4个NSF软件研究所，每个研究所都支持许多NSF资助的研究人员，以提供FuncX的用例，塑造其设计，并评估其实施。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

DLHub: Simplifying publication, discovery, and use of machine learning models in science

• DOI: 10.1016/j.jpdc.2020.08.006
• 发表时间: 2021-01-01
• 期刊: JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING
• 影响因子: 3.8
• 作者: Li, Zhuozhao;Chard, Ryan;Foster, Ian
• 通讯作者: Foster, Ian

Enhancing Automated FaaS with Cost-aware Provisioning of Cloud Resources

通过具有成本意识的云资源配置来增强自动化 FaaS

• DOI: 10.1109/escience51609.2021.00053
• 发表时间: 2021
• 期刊: 2021 IEEE 17th International Conference on eScience (eScience
• 作者: Baughman, Matt;Foster, Ian;Chard, Kyle
• 通讯作者: Chard, Kyle

? unc X: Federated Function as a Service for Science

？

• DOI: 10.1109/tpds.2022.3208767
• 发表时间: 2022
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Li, Zhuozhao;Chard, Ryan;Babuji, Yadu;Galewsky, Ben;Skluzacek, Tyler J.;Nagaitsev, Kirill;Woodard, Anna;Blaiszik, Ben;Bryan, Josh;Katz, Daniel S.
• 通讯作者: Katz, Daniel S.

A Serverless Framework for Distributed Bulk Metadata Extraction

用于分布式批量元数据提取的无服务器框架

• DOI: 10.1145/3431379.3460636
• 发表时间: 2021
• 期刊: Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing (HPDC
• 作者: Skluzacek, Tyler J.;Wong, Ryan;Li, Zhuozhao;Chard, Ryan;Chard, Kyle;Foster, Ian
• 通讯作者: Foster, Ian

An Empirical Study of Package Dependencies and Lifetimes in Binder Python Containers

Binder Python 容器中包依赖关系和生命周期的实证研究

• DOI: 10.1109/escience51609.2021.00032
• 发表时间: 2021
• 期刊: 2021 IEEE 17th International Conference on eScience (eScience
• 作者: Shaffer, Tim;Chard, Kyle;Thain, Douglas
• 通讯作者: Thain, Douglas