CAREER: Harnessing Serverless Functions to Build Highly Elastic Cloud Storage Infrastructure

职业：利用无服务器功能构建高弹性的云存储基础设施

• 批准号: 2322860
• 负责人: Yue Cheng
• 金额: $ 57.29万
• 依托单位: University of Virginia Main Campus
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2026-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2322860&HistoricalAwards=false
• 关键词: CAREER; Harnessing; Serverless; Functions; Build

Crucial data-intensive applications such as big data analytics, artificial intelligence, and scientific computing exhibit highly dynamic and heterogeneous data access patterns. These applications require a large data storage system that is fast, elastic, cost-effective, and easy-to-use. Unfortunately, current cloud storage offerings are unable to satisfy all of these requirements: big data analytics applications can run up to 500X slower when they run as a cloud-native application using a cloud object store; provisioning more cloud storage resources does not solve the performance problem and storage over-provisioning leads to millions of dollars of extra cost. This project rethinks the design and implementation of next-generation, cloud-native storage infrastructures via a new approach. This approach is enabled by a novel take on the serverless computing paradigm -- serverless functions are elastic and truly pay-per-use, making them a fast and inexpensive storage medium. This project aims to build InfiniStore, a new cloud storage system, which elastically scales in response to constantly changing data access patterns by orchestrating the collective memory of serverless functions. Following an end-to-end, stacked approach, this project will develop an elastic and inexpensive memory store using serverless functions, techniques that provide strong consistency guarantees for data replicated in serverless functions, fault tolerance mechanisms to maximize the data availability in highly unreliable serverless environments, and a scalable serverless file metadata service to support file semantics.This project has the potential to redefine cloud storage pricing models -- InfiniStore enables memory-store-level pay-per-access that lets cloud users pay for the data they access. This project will enable, in a broad range of disciplines, the development of new, stateful, serverless applications that have not been previously possible. All artifacts will be open source to support the use and development both in academia and industry. This project has an integrated educational plan, which also harnesses serverless computing but uses it to develop a new serverless notebook cloud service, called InfiniCloud, for interdisciplinary education. InfiniCloud will allow educators and students to write and run serial or parallel Python programs at any scale, without needing to manage servers or clusters. Moreover, this project will use InfiniStore and InfiniCloud as a basis for various educational activities including curriculum development, diversity building, undergraduate engagement, and workshops.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

重要的数据密集型应用程序，例如大数据分析，人工智能和科学计算，表现出高度动态和异质的数据访问模式。这些应用程序需要快速，弹性，成本效益且易于使用的大型数据存储系统。不幸的是，当前的云存储产品无法满足所有这些要求：大数据分析应用程序使用云对象存储作为云本地应用程序运行时，应用程序的运行速度最高可慢500倍；提供更多的云存储资源不能解决性能问题，并且存储过度提供会导致数百万美元的额外成本。该项目通过新方法重新考虑了下一代，云原生存储基础架构的设计和实施。通过对无服务器计算范式进行新颖的方法来启用这种方法 - 无服务器功能是弹性且真正每次使用的付费，使其成为快速且廉价的存储介质。该项目旨在构建一种新的云存储系统Infinistore，该系统通过策划无服务器功能的集体内存来响应不断变化的数据访问模式而弹性地扩展。遵循端到端的堆叠方法，该项目将使用无服务器功能开发弹性且廉价的内存存储店，这些功能可为在无服务器功能中复制的数据提供强大的一致性保证，可容忍的可容忍机制，可最大程度地提高高度不可靠的无服务器环境中的数据可用性，并提供无用的specter files files files files filef files filef to ins this ins intim ins this ins intim ins intim ins ins inter。启用记忆储存级别的按付费付费，使云用户可以为其访问的数据付费。该项目将在广泛的学科中启用，开发了以前无法实现的新，状态，无服务的应用程序。所有工件将是支持学术界和行业使用和开发的开源。该项目具有集成的教育计划，该计划还可以利用无服务器计算，但使用它来开发一种新的无服务器笔记本云服务，称为Infinicloud，用于跨学科教育。 Infinicloud将允许教育者和学生在任何规模上编写和运行串行或并行的Python程序，而无需管理服务器或集群。此外，该项目将使用Infinistore和Infinicloud作为各种教育活动的基础，包括课程发展，多样性建设，本科参与度和讲习班。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估来通过评估来获得支持的。

项目成果

?FS: A Scalable and Elastic Distributed File System Metadata Service using Serverless Functions

?FS：使用无服务器功能的可扩展且弹性的分布式文件系统元数据服务

• 发表时间: 2024
• 期刊: ACM ASPLOS 2023
• 作者: Carver, Benjamin;Han, Runzhou;Zhang, Jingyuan;Zheng, Mai;Cheng, Yue
• 通讯作者: Cheng, Yue

InfiniStore: Elastic Serverless Cloud Storage

InfiniStore：弹性无服务器云存储

• DOI: 10.14778/3587136.3587139
• 发表时间: 2023
• 期刊: Proceedings of the VLDB Endowment
• 影响因子: 2.5
• 作者: Zhang, Jingyuan;Wang, Ao;Ma, Xiaolong;Carver, Benjamin;Newman, Nicholas John;Anwar, Ali;Rupprecht, Lukas;Tarasov, Vasily;Skourtis, Dimitrios;Yan, Feng
• 通讯作者: Yan, Feng

SHADE: Enable Fundamental Cacheability for Distributed Deep Learning Training

• 发表时间: 2023
• 作者: Redwan Ibne Seraj Khan;Ahmad Hossein Yazdani;Yuqi Fu;Arnab K. Paul;Bo Ji;Xun Jian;Yue Cheng;A. R. Butt