Collaborative Research: NGS: Dynamic Runtime and Compilation Support for I/O-Intensive Applications

摘要：CNS-0406341万亿级高性能计算使科学家能够解决非常大的和计算上具有挑战性的问题，例如在科学计算领域中发现的问题。 然而，随着计算扩展到前所未有的水平，它也变得非常数据密集和I/O密集。因此，I/O正在成为一个主要的瓶颈，从而减缓了科学发现和数据分析的预期速度。此外，为了科普更大的问题和数据量，模型和应用程序正在被设计为动态的。也就是说，应用程序在其计算模式和数据访问模式方面都是动态的。由于系统和应用程序的复杂性，因此，解决研究问题并在运行时系统和编译器级别开发动态技术以按正确比例扩展I/O是非常重要的。该项目的目标是设计和开发下一代软件技术，以解决大规模科学应用的数据，I/O和存储瓶颈。特别是，这个项目的目的是研究动态运行时和编译技术的可扩展的I/O优化的大型系统。另一个重要的方面是通过学习和表征I/O和数据访问的性能来驱动这些优化，并随后使用它们来开发动态运行时和编译系统使用的规则，以实现高性能I/O。目前最先进的编译器对I/O密集型应用程序的支持非常缺乏。许多大规模I/O密集型应用程序的可扩展性需求可以从健壮的动态编译和链接基础设施中受益匪浅。该项目的具体目标是：了解I/O密集型应用程序动态变化的数据访问需求。在元数据管理器中捕获运行时的动态访问模式和应用程序引导信息。设计并实现了基于运行时访问模式和元数据管理器收集的性能统计数据的动态编译技术。设计并实现一种布局管理器，该管理器从多个同时执行的应用程序收集存储格式（布局）建议，并为磁盘驻留数据和磁带驻留数据确定全局可接受的存储布局。设计并实现了一个高级的、动态的、易于使用的I/O库，它可以被动态编译器/链接器调用。研究什么类型的用户指定的提示可以传递给运行时系统/编译器，以及如何将它们合并以减少与动态编译相关的开销。在实际的I/O密集型工作负载下评估所开发的动态编译/链接基础架构的性能，并量化与动态编译相关的运行时开销。在公共领域提供已开发的基础设施和实验结果，并将研究结果纳入本科生和研究生课程。

Abstract, CNS-0406341Tera-scale high-performance computing has enabled scientists to tackle very large and computationally challenging problems, such as those found in the scientific computing domain. However, as computing scales to levels never seen before, it also becomes extremely data intensive and I/O intensive. Thus, I/O is becoming a major bottleneck, thereby slowing the expected pace of scientific discovery and analysis of data. Furthermore, in order to cope with larger problems and data sizes, models and applications are being designed to be dynamic in nature. That is, the applications are dynamic both in terms of their computation patterns as well as data access patterns. Due to the complexities of systems and applications, it is, therefore, very important to address research issues and develop dynamic techniques at the level of runtime systems and compilers to scale I/O in the right proportions. Technical Merit:The objectives of this project are to design and develop next generation software techniques to address the data, I/O, and storage bottlenecks for large-scale scientific applications. Particularly, this project aims to investigate dynamic runtime and compilation techniques for scalable I/O optimizations for largescale systems. Another important aspect will be to drive these optimizations by learning and characterizing performance of I/O and data accesses, and subsequently using those to develop rules that will be used by dynamic runtime and compilation systems to enable high-performance I/O. Current state-of-the-art compiler support for I/O-intensive applications is tremendously lacking. Runtime needs of many large-scale I/O-intensive applications can benefit a lot from a robust dynamic compilation and linking infrastructure. The specific objectives of this project are:. Developing an understanding of dynamically varying data access needs of I/O-intensive applications,. Capturing dynamic access patterns and application steering information at runtime within a metadata manager,. Designing and implementing dynamic compilation techniques based on the runtime access patternsand performance statistics collected by the metadata manager,. Designing and implementing a layout manager that collects storage format (layout) suggestions from multiple concurrently executing applications and determines the globally acceptable storage layouts for disk-resident and tape-resident data,. Designing and implementing a high-level, dynamic, easy-to-use I/O library that can be invoked by the dynamic compiler/linker,. Investigating what types of user-specified hints can be passed to the runtime system/compiler, andhow they can be incorporated to reduce the overheads associated with dynamic compilation,. Evaluating the performance of the developed dynamic compilation/linking infrastructure underrealistic I/O-intensive workloads and quantifying the runtime overheads associated with dynamiccompilation, and. Providing the developed infrastructure and experimental findings in the public domain, andincorporating the research findings into the undergraduate and graduate curriculum.