CAREER: Harnessing Serverless Functions to Build Highly Elastic Cloud Storage Infrastructure

大数据分析、人工智能和科学计算等关键数据密集型应用表现出高度动态和异构性的数据访问模式。这些应用需要一个快速、灵活、经济实惠且易于使用的大型数据存储系统。遗憾的是，当前的云存储产品无法满足所有这些要求：当大数据分析应用程序作为使用云对象存储的云本地应用程序运行时，它们的运行速度可能会减慢500倍；调配更多的云存储资源并不能解决性能问题，存储过度调配会导致数百万美元的额外成本。该项目通过一种新方法重新思考下一代云本地存储基础架构的设计和实施。这种方法是由一种新的无服务器计算范例实现的--无服务器功能是灵活的，真正的按使用付费，使它们成为一种快速且廉价的存储介质。该项目旨在构建InfiniStore，这是一个新的云存储系统，通过协调无服务器功能的集体内存，灵活地扩展以响应不断变化的数据访问模式。按照端到端、堆叠的方法，该项目将开发一个弹性和廉价的内存存储，使用无服务器功能，技术为在无服务器功能中复制的数据提供强大的一致性保证，容错机制，以最大限度地提高高度不可靠的无服务器环境中的数据可用性，以及可扩展的无服务器文件元数据服务，以支持文件语义。该项目有可能重新定义云存储定价模型-InfiniStore启用内存存储级别的按访问付费，允许云用户为他们访问的数据付费。该项目将在广泛的学科范围内支持开发新的、有状态的、无服务器的应用程序，这是以前不可能实现的。所有的文物都将是开源的，以支持学术界和工业界的使用和开发。该项目有一个集成的教育计划，该计划也利用无服务器计算，但使用它来开发一种新的无服务器笔记本云服务，称为InfiniCloud，用于跨学科教育。InfiniCloud将允许教育工作者和学生编写和运行任何规模的串行或并行Python程序，而不需要管理服务器或集群。此外，该项目将使用InfiniStore和InfiniCloud作为各种教育活动的基础，包括课程开发、多样性建设、本科生参与和工作室。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Crucial data-intensive applications such as big data analytics, artificial intelligence, and scientific computing exhibit highly dynamic and heterogeneous data access patterns. These applications require a large data storage system that is fast, elastic, cost-effective, and easy-to-use. Unfortunately, current cloud storage offerings are unable to satisfy all of these requirements: big data analytics applications can run up to 500X slower when they run as a cloud-native application using a cloud object store; provisioning more cloud storage resources does not solve the performance problem and storage over-provisioning leads to millions of dollars of extra cost. This project rethinks the design and implementation of next-generation, cloud-native storage infrastructures via a new approach. This approach is enabled by a novel take on the serverless computing paradigm -- serverless functions are elastic and truly pay-per-use, making them a fast and inexpensive storage medium. This project aims to build InfiniStore, a new cloud storage system, which elastically scales in response to constantly changing data access patterns by orchestrating the collective memory of serverless functions. Following an end-to-end, stacked approach, this project will develop an elastic and inexpensive memory store using serverless functions, techniques that provide strong consistency guarantees for data replicated in serverless functions, fault tolerance mechanisms to maximize the data availability in highly unreliable serverless environments, and a scalable serverless file metadata service to support file semantics.This project has the potential to redefine cloud storage pricing models -- InfiniStore enables memory-store-level pay-per-access that lets cloud users pay for the data they access. This project will enable, in a broad range of disciplines, the development of new, stateful, serverless applications that have not been previously possible. All artifacts will be open source to support the use and development both in academia and industry. This project has an integrated educational plan, which also harnesses serverless computing but uses it to develop a new serverless notebook cloud service, called InfiniCloud, for interdisciplinary education. InfiniCloud will allow educators and students to write and run serial or parallel Python programs at any scale, without needing to manage servers or clusters. Moreover, this project will use InfiniStore and InfiniCloud as a basis for various educational activities including curriculum development, diversity building, undergraduate engagement, and workshops.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.