Storage and Memory Management for Data-Intensive Computing

随着云计算模式的广泛采用，计算正越来越多地进入数据中心，而数据中心计算本身也正在因超大容量主存储器的可用性、具有远程直接存储器访问(RDMA)功能的Infiniband联网和高容量固态硬盘(SSD)存储等新硬件技术的商品化而发生根本性变化。在此环境中，计算节点机架已成为新的数据中心构建块，称为机架规模计算。此外，随着英特尔3D XPoint技术的引入，非易失性存储器正在成为商业现实。随着旧的性能瓶颈被消除，新的瓶颈出现，这场硬件革命对系统级软件，特别是存储和内存管理层产生了深远的影响。*硬件进步和分布式数据处理框架正在推动数据分析和机器学习的快速发展，使自动语言翻译、图像理解和自动驾驶车辆(仅举几例)中的应用成为可能。今天的框架通常是可伸缩的，但效率非常低，浪费了新硬件的潜力，软件管理费用过高。我的研究探索了管理迅速增长的硬件资源财富的方法，使未来的数据密集型分析应用程序能够在更大规模、更高效和更可靠的情况下运行。我们总结了实现这一目标的三个具体途径。*1.统一机架规模的永久内存存储系统：现有的永久内存存储研究改进了为慢速、基于块的设备设计的传统文件系统，但在扩展到单个节点之外时，它没有考虑分配、访问控制和一致性问题。未来的机架规模计算将需要一个可扩展、并行和分布式的文件系统，以展示整个节点机架上新出现的永久内存的性能。*2.数据分析框架的内存局部性感知调度：许多大型数据分析应用程序使用的是ApacheSpark或类似的计算框架，它们的设计假设数据存储在访问速度较慢的磁盘上。当机架中的节点通过高速网络连接时，磁盘位置变得越来越不重要。我们正在探索利用有关输入数据的哪些部分已在每个节点的内存中的信息的方法，从而实现更智能的任务布置。*3.内存中分布式时空数据分析：传感器和配备GPS的移动设备的兴起正在推动时空数据的快速增长，使需要复杂时空数据分析的新型应用成为可能。然而，不断扩大的数据量意味着这些数据集可能很快就会超过大内存服务器的容量，这需要新的内存机架规模处理技术。

Computation is increasingly moving into data centres with the widespread adoption of the Cloud computing model, while data centre computing itself is being radically transformed by the availability of very large main memories, the commodification of new hardware technologies such as Infiniband networking with Remote Direct Memory Access (RDMA) capabilities, and high capacity solid state disk (SSD) storage. In this environment, a rack of compute nodes has become the new data centre building block, known as rack-scale computing. Also, non-volatile memory is becoming a commercial reality with the introduction of Intel's 3D XPoint technology. This hardware revolution has profound implications for systems-level software, especially the storage and memory management layers, as old performance bottlenecks are eliminated and new ones emerge. ***Hardware advances and distributed data processing frameworks are fueling rapid advances in data analytics and machine learning, enabling applications in automated language translation, image understanding and autonomous vehicles (to name just a few). Today's frameworks are often scalable but highly inefficient, squandering the potential of new hardware with excessive software overheads. My research explores ways to manage the burgeoning wealth of hardware resources, enabling future data-intensive analytics applications to operate at larger scales, more efficiently, and with greater reliability. We summarize three specific avenues to achieve this goal.***1. A unified rack-scale persistent memory storage system: Existing research on persistent memory storage improves on traditional file systems designed for slow, block-based devices, but it does not consider issues of allocation, access control, and consistency when scaling beyond a single node. Future rack-scale computing will demand a scalable, parallel and distributed file system that exposes the performance of emerging persistent memory across a full rack of nodes. ***2. Memory-locality aware scheduling for data analytics frameworks: Many large-scale data analytics applications use Apache Spark or similar computing frameworks, which are designed with the assumption that data is stored on slow-to-access disks. Disk locality is of decreasing importance when nodes in a rack are connected by high-speed networks. We are exploring ways to exploit information about what parts of the input data are already in memory at each node, making smarter task placement possible. ***3. In-memory distributed spatiotemporal data analytics: The rise of sensors and GPS-equipped mobile devices is driving rapid growth of spatiotemporal data, enabling novel applications that require sophisticated spatiotemporal data analytics. The expanding data volume means these datasets may soon exceed the capacity of even large-memory servers, however, requiring new techniques for in-memory rack-scale processing.