Collaborative Research: NGS: Dynamic Runtime and Compilation Support for I/O-Intensive Applications

摘要：cns - 0406341tb级高性能计算使科学家能够解决非常大的和具有计算挑战性的问题，例如科学计算领域的问题。然而，随着计算规模达到前所未有的水平，它也变得非常数据密集和I/O密集。因此，I/O正在成为一个主要的瓶颈，从而减慢了科学发现和数据分析的预期速度。此外，为了处理更大的问题和数据规模，模型和应用程序被设计成本质上是动态的。也就是说，应用程序在计算模式和数据访问模式方面都是动态的。由于系统和应用程序的复杂性，因此，在运行时系统和编译器级别上解决研究问题和开发动态技术以适当的比例扩展I/O是非常重要的。技术优点：该项目的目标是设计和开发下一代软件技术，以解决大规模科学应用的数据、I/O和存储瓶颈。特别地，这个项目旨在研究用于大规模系统的可伸缩I/O优化的动态运行时和编译技术。另一个重要方面是通过学习和描述I/O和数据访问的性能来驱动这些优化，然后使用这些来开发规则，这些规则将被动态运行时和编译系统使用，以启用高性能I/O。当前最先进的编译器对I/ o密集型应用程序的支持非常缺乏。许多大规模I/ o密集型应用程序的运行时需求可以从健壮的动态编译和链接基础设施中获益良多。该项目的具体目标是：。理解I/ o密集型应用程序动态变化的数据访问需求。在元数据管理器中，在运行时捕获动态访问模式和应用程序转向信息。基于运行时访问模式和元数据管理器收集的性能统计信息，设计和实现动态编译技术。设计和实现一个布局管理器，它从多个并发执行的应用程序收集存储格式（布局）建议，并确定磁盘驻留和磁带驻留数据的全局可接受的存储布局。设计和实现一个高级的、动态的、易于使用的I/O库，它可以被动态编译器/链接器调用。研究哪些类型的用户指定提示可以传递给运行时系统/编译器，以及如何将它们合并以减少与动态编译相关的开销。评估开发的动态编译/链接基础架构在实际I/ o密集型工作负载下的性能，并量化与动态编译相关的运行时开销，以及。在公共领域提供发达的基础设施和实验成果，并将研究成果纳入本科和研究生课程。

Abstract, CNS-0406341Tera-scale high-performance computing has enabled scientists to tackle very large and computationally challenging problems, such as those found in the scientific computing domain. However, as computing scales to levels never seen before, it also becomes extremely data intensive and I/O intensive. Thus, I/O is becoming a major bottleneck, thereby slowing the expected pace of scientific discovery and analysis of data. Furthermore, in order to cope with larger problems and data sizes, models and applications are being designed to be dynamic in nature. That is, the applications are dynamic both in terms of their computation patterns as well as data access patterns. Due to the complexities of systems and applications, it is, therefore, very important to address research issues and develop dynamic techniques at the level of runtime systems and compilers to scale I/O in the right proportions. Technical Merit:The objectives of this project are to design and develop next generation software techniques to address the data, I/O, and storage bottlenecks for large-scale scientific applications. Particularly, this project aims to investigate dynamic runtime and compilation techniques for scalable I/O optimizations for largescale systems. Another important aspect will be to drive these optimizations by learning and characterizing performance of I/O and data accesses, and subsequently using those to develop rules that will be used by dynamic runtime and compilation systems to enable high-performance I/O. Current state-of-the-art compiler support for I/O-intensive applications is tremendously lacking. Runtime needs of many large-scale I/O-intensive applications can benefit a lot from a robust dynamic compilation and linking infrastructure. The specific objectives of this project are:. Developing an understanding of dynamically varying data access needs of I/O-intensive applications,. Capturing dynamic access patterns and application steering information at runtime within a metadata manager,. Designing and implementing dynamic compilation techniques based on the runtime access patternsand performance statistics collected by the metadata manager,. Designing and implementing a layout manager that collects storage format (layout) suggestions from multiple concurrently executing applications and determines the globally acceptable storage layouts for disk-resident and tape-resident data,. Designing and implementing a high-level, dynamic, easy-to-use I/O library that can be invoked by the dynamic compiler/linker,. Investigating what types of user-specified hints can be passed to the runtime system/compiler, andhow they can be incorporated to reduce the overheads associated with dynamic compilation,. Evaluating the performance of the developed dynamic compilation/linking infrastructure underrealistic I/O-intensive workloads and quantifying the runtime overheads associated with dynamiccompilation, and. Providing the developed infrastructure and experimental findings in the public domain, andincorporating the research findings into the undergraduate and graduate curriculum.