CRII: CSR: Large-scale Systems Software Atop Scale-out In-memory Storage

在过去5年中，对内存中数据中心存储的研究已经产生了能够提供每秒数十亿条信息的按需访问的系统。然而，这些快速系统的好处并没有为用户带来更强大和更有趣的应用程序。关键问题是，这些新的存储系统被精简以提供速度，并且只能满足非常基本的信息请求。为了弥补这一点，应用程序必须向这些存储系统发出多次请求，以执行它们能够通过对较旧的数据库系统的单个请求执行的相同操作。其结果是，应用程序花费了它们从这些系统获得的大部分性能优势来克服其有限的编程模型。这项工作试图了解这些大规模内存中键值存储的有限接口的隐藏成本，并使用唯一适合于这些系统的新的存储过程模型来降低这些成本。为了探索这些成本，今天将以三个不同领域的数据密集型应用程序为原型-S低延迟键值存储，首先使用它们的传统接口，然后再次使用内置到存储系统中的自定义应用程序逻辑。这一探索的结果将激励和帮助设计这项工作的主要贡献：在键值存储中安全地配置应用程序逻辑的通用模型。应用原型将包括横向扩展数据库内核、简单的横向扩展图形数据库和大规模机器学习算法。用于开发大型软件系统的丰富编程模型将降低构建信息密集型应用程序的门槛：这是一门需要专业知识的艺术。配备快速存储过程的低延迟系统将成为在工业、科学和军事领域构建新的数据密集型系统的强大工具。它们将实现对社交和自然图表的深入实时分析；数百万用户在丰富的互动世界中操纵和修改共享环境；在大都市地区的高速公路上对数百万辆自动驾驶车辆进行细粒度的协调和路线选择；以及为军事情报提供实时决策支持。所有组件都将构建为一个实用和可用的软件系统，所有开发工作都将公开进行，并以开放源码的形式提供。

Research over the past 5 years on in-memory data center storage has resulted in systems that can provide on-demand access to billions of pieces of information per second. However, the benefits of these fast systems has not resulted in more powerful and interesting applications for users. The key problem is that these new storage systems are stripped down to provide speed and can only fulfill very basic requests for information. To compensate, applications must make many requests to these storage systems to perform the same operations they were able to perform with a single request to older database systems. The result is that applications spend most of the performance benefits they gain from these systems overcoming their limited programming model.This work seeks to understand the hidden costs of the limited interfaces of these large-scale in-memory key-value stores and to alleviate those costs with a new stored procedure model uniquely suited to these systems. To explore these costs, data-intensive applications from three different domains will be prototyped on today?s low-latency key-value stores, first with their conventional interfaces and then again with custom application logic built into the storage system. The results of this exploration will motivate and aid in the design of the main contribution of this work: a generalized model for safely collocating application logic inside of key-value stores. Application prototypes will include a scale-out database kernel, a simple scale-out graph database, and large-scale machine learning algorithms.A rich programming model for developing large-scale software systems will lower the barrier for building information-intensive applications: an art that requires specialized expertise. Low-latency systems equipped with fast stored procedures will be a powerful tool for building new, data-intensive systems in industry, science, and the military. They will enable deep real-time analysis of social and natural graphs; rich, interactive worlds where millions of users manipulate and modify a shared environment; the fine-grained coordination and routing of millions of autonomous vehicles on the highways in a metropolitan area; and real-time decision support for military intelligence. All components will be built as a practical and usable software system with all development done publicly and available as open-source.