SHF: EAGER: HI-HDFS - Holistic I/O optimizations for the Hadoop distributed filesystem

文件系统及其过时的POSIX“字节流”接口与当今应用程序的通用I/O需求存在阻抗不匹配的问题。具体来说，从应用程序到原始存储设备的I/O路径变得越来越长，并且涉及到复杂的软件和硬件组件的相互作用。这就产生了复杂的聚合I/O模式，应用程序开发人员（通常是对大规模并发性如何造成I/O瓶颈知之甚少的主题专家）无法仅凭直觉对其进行优化。标榜其高可伸缩性的文件系统，比如Hadoop分布式文件系统，很大程度上是通过将应用程序限制为顺序访问模式来实现的。对于具有复杂数据模型的分析应用程序，是否可以加速Hadoop分布式文件系统的I/O性能，这些数据模型不能轻易地连续序列化数据以实现快速顺序访问，这个问题仍然没有解决。该项目旨在解决这个问题，并构建HI-HDFS——一个自动收集和管理语义更丰富的I/O元数据的框架，以指导Hadoop分布式文件系统中的对象放置。HI-HDFS框架在一个可导航的图结构中综合了整个数据中心的软件组件的I/O活动，以识别I/O活动中与应用程序无关的主题。一种新的I/O预测技术通过检查小规模运行的I/O活动来识别和改善大规模的瓶颈。总的来说，HI-HDFS框架挑战了手动数据放置是I/O性能基石的I/O优化口号，并为高性能计算机的下一代以对象为中心的存储系统铺平了道路。将在紧急反应领域的复杂数据处理工作量上审查这种自动化方法的效力，该领域显示了作为现代分析应用特征的输入/输出模式。这项工作预计将产生更广泛的影响，包括开源原型实现，以及对俄亥俄州立大学计算机科学和数据分析本科专业云计算课程的教学影响。

File systems and their outdated POSIX "byte stream" interface suffer from an impedance mismatch with the versatile I/O requirements of today's applications. Specifically, the I/O path from the application to the raw storage device is becoming longer and it involves the interplay of intricate software and hardware components. This produces complex aggregate I/O patterns that application developers (often subject matter experts with limited knowledge of how massive concurrency creates I/O bottlenecks) cannot optimize based on intuition alone. File systems that tout their high scalability, such as the Hadoop distributed file system, largely do so by limiting applications to sequential access patterns. The question of whether one can accelerate the I/O performance of the Hadoop distributed file system for analytical applications with complex data models that cannot readily serialize data contiguously for fast sequential access remains open. This project seeks to address this question and build HI-HDFS -- a framework that automatically collects and manages semantically richer I/O metadata to guide object placement in the Hadoop distributed file system. The HI-HDFS framework synthesizes the I/O activity across software components throughout the datacenter in a navigable graph structure to identify application-agnostic motifs in I/O activity. A novel I/O forecasting technique identifies and ameliorates bottlenecks at large scale by inspecting I/O activity from small-scale runs. Overall, the HI-HDFS framework challenges the I/O optimization mantra that manual data placement is the cornerstone of I/O performance and paves the way towards next-generation object-centric storage systems for high-performance computers. The efficacy of this automated approach will be examined on a complex data processing workload from the domain of emergency response which exhibits I/O patterns that are characteristic of modern analytical applications. The broader impacts of this work are expected to include open-source prototype implementations as well as pedagogical impact on a cloud computing course for both Computer Science and Data Analytics undergraduate majors at Ohio State.