CAREER: Harnessing Serverless Functions to Build Highly Elastic Cloud Storage Infrastructure

大数据分析、人工智能和科学计算等关键数据密集型应用呈现出高度动态和异构的数据访问模式。这些应用程序需要一个快速、弹性、经济高效且易于使用的大型数据存储系统。不幸的是，目前的云存储产品无法满足所有这些要求：大数据分析应用程序在使用云对象存储作为云原生应用程序运行时，运行速度可能会慢500倍;配置更多的云存储资源并不能解决性能问题，存储过度配置会导致数百万美元的额外成本。该项目通过一种新的方法重新思考下一代云原生存储基础架构的设计和实施。这种方法是由一种新的无服务器计算模式实现的--无服务器功能是弹性的，真正的按使用付费，使它们成为一种快速而廉价的存储介质。该项目旨在构建一个新的云存储系统InfiniStore，该系统通过编排无服务器功能的集体内存来弹性扩展，以响应不断变化的数据访问模式。遵循端到端的堆叠方法，该项目将使用无服务器功能开发一种弹性和廉价的内存存储，为无服务器功能中复制的数据提供强一致性保证的技术，容错机制，以最大限度地提高高度不可靠的无服务器环境中的数据可用性，和一个可扩展的无服务器文件元数据服务，以支持文件语义。该项目有可能重新定义云存储定价模型-- InfiniStore使内存存储-按访问付费级别，让云用户为他们访问的数据付费。该项目将在广泛的学科中实现新的、有状态的、无服务器的应用程序的开发，这在以前是不可能的。所有工件都将是开源的，以支持学术界和工业界的使用和开发。该项目有一个集成的教育计划，它也利用无服务器计算，但使用它来开发一个新的无服务器笔记本云服务，称为InfiniCloud，用于跨学科教育。InfiniCloud将允许教育工作者和学生以任何规模编写和运行串行或并行Python程序，而无需管理服务器或集群。此外，该项目将使用InfiniStore和InfiniCloud作为各种教育活动的基础，包括课程开发，多样性建设，本科生参与和研讨会。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Crucial data-intensive applications such as big data analytics, artificial intelligence, and scientific computing exhibit highly dynamic and heterogeneous data access patterns. These applications require a large data storage system that is fast, elastic, cost-effective, and easy-to-use. Unfortunately, current cloud storage offerings are unable to satisfy all of these requirements: big data analytics applications can run up to 500X slower when they run as a cloud-native application using a cloud object store; provisioning more cloud storage resources does not solve the performance problem and storage over-provisioning leads to millions of dollars of extra cost. This project rethinks the design and implementation of next-generation, cloud-native storage infrastructures via a new approach. This approach is enabled by a novel take on the serverless computing paradigm -- serverless functions are elastic and truly pay-per-use, making them a fast and inexpensive storage medium. This project aims to build InfiniStore, a new cloud storage system, which elastically scales in response to constantly changing data access patterns by orchestrating the collective memory of serverless functions. Following an end-to-end, stacked approach, this project will develop an elastic and inexpensive memory store using serverless functions, techniques that provide strong consistency guarantees for data replicated in serverless functions, fault tolerance mechanisms to maximize the data availability in highly unreliable serverless environments, and a scalable serverless file metadata service to support file semantics.This project has the potential to redefine cloud storage pricing models -- InfiniStore enables memory-store-level pay-per-access that lets cloud users pay for the data they access. This project will enable, in a broad range of disciplines, the development of new, stateful, serverless applications that have not been previously possible. All artifacts will be open source to support the use and development both in academia and industry. This project has an integrated educational plan, which also harnesses serverless computing but uses it to develop a new serverless notebook cloud service, called InfiniCloud, for interdisciplinary education. InfiniCloud will allow educators and students to write and run serial or parallel Python programs at any scale, without needing to manage servers or clusters. Moreover, this project will use InfiniStore and InfiniCloud as a basis for various educational activities including curriculum development, diversity building, undergraduate engagement, and workshops.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.