Collaborative Research: Compiler-Supported Simulation of Scalable Applications for Wide-Area Distributed Computing Systems

广域分布式计算系统代表了科学和商业计算的未来。这样的系统将实现广泛的未来应用，具有巨大的经济和社会影响应用的潜力，如分布式多媒体服务、基于网络的协作、分布式超级计算和远程沉浸。然而，在实现这一愿景之前，需要在软件系统设计、实现和评估的几乎所有方面进行广泛的研究。离散事件仿真已经成为传统计算系统和应用设计和分析的重要工具。然而，由于必须模拟的软件、硬件和网络子系统的规模，在广域分布式系统上执行应用程序的实际仿真是一项具有挑战性的任务。然而，这些组件的复杂性及其复杂的闭环相互作用需要对组件及其相互作用进行足够详细的建模，以适当地预测它们对整体系统性能的影响。在最近的工作中，pi合作获得了一些令人兴奋的初步结果，表明特定的编译器信息可以极大地提高消息传递程序模拟的效率和可伸缩性。例如，研究表明，通过使用并行模拟和本提案主题的编译器优化类型，可以比实时更快地模拟在多达128个处理器上执行的可扩展的ASCI内核基准应用程序SWEEP3D。此外，编译器优化的模拟可以在数千个处理器上评估非常大的数据集：可以在多达10,000个处理器上模拟4000万个问题大小的Sweep3D的性能。通过使用编译器信息，可能有许多其他策略可以显著提高仿真的可伸缩性和性能，但迄今为止还没有对这些策略进行过研究。需要一个全面的研究计划来开发它们的潜力并评估它们对模拟现实世界应用的影响。当前提案的重点是开发基于编译器的技术，以提高并行离散事件模拟的效率，并使用这些技术来评估广域分布式系统的应用程序和系统软件性能。这一提议有三个关键组成部分：1。探索一系列编译器支持的策略，以高效地模拟动态、大规模的系统和应用。通过扩展我们现有的编译器和仿真基础设施，使用这些策略开发适用于广域分布式系统的实用性能工具。这需要解决由这些系统的动态特性和缺乏良好定义的度量来衡量在这种环境中有效的应用级性能所带来的额外挑战。评估这些策略在真实世界分布式应用程序中的有效性，如SF-Express、大规模分布式交互模拟（DIS）环境和分布式视频点播服务器（分布式多媒体应用程序）。拟议的研究计划建立在pi研究小组之间数年的合作基础上，并汇集了大规模并行程序并行模拟，广域网并行模拟以及并行编译器及其用于支持性能评估的关键优势。这一研究项目还补充了针对开发操作系统服务（例如，Globus、Legion和WebOS）和编程环境（例如，Legion、Globe和grad）的广域系统正在进行的软件工作。因此，拟议的研究计划代表了成功的广域分布式系统开发和部署的软件支持的必不可少的第三条腿。

Wide-area distributed computing systems represent the future of scientific and commercial computing.Such systems will enable a wide range of futuristic applications with the potential for enormous economicand social impact applications such as distributed multimedia services, Web-based collaboration,distributed supercomputing, and teleimmersion. Before this vision can be realized, however, extensiveresearch will be required in virtually all aspects of software system design, implementation, and evaluation.Discrete-event simulation has been an essential tool for the design and analysis of traditional computationalsystems and applications. Realistic simulation of applications executing on wide-area distributed systems,however, is a challenging task because of the scale of the software, hardware and network subsystems thatmust be simulated. Nevertheless, the intricacy of these components and their complex, closed-loopinteractions require the components and their interactions to be modeled in sufficient detail to appropriatelypredict their impact on overall system performance.In recent work, the PIs have collaboratively obtained some exciting but preliminary results showing thatspecific compiler information can greatly enhance the efficiency and scalability of simulation of message-passing programs. For instance, it was shown that simulation of a scalable ASCI kernel benchmarkapplication called SWEEP3D executing on up to 128 processors could be simulated faster than real-time byusing parallel simulations together with the type of compiler optimizations that are the subject of thisproposal. Also, the compiler-optimized simulation can evaluate very large data sets on thousands ofprocessors: it was possible to simulate the performance of a 40 million-problem size Sweep3D for up to10,000 processors. There are potentially a number of other strategies to dramatically improve simulationscalability and performance by using compiler information, none of which have been studied so far. Acomprehensive program of research is required to develop their potential and evaluate their impact onsimulation of real world applications.The focus of the current proposal is to develop compiler-based techniques for improving the efficiency ofparallel discrete event simulation, and to use these techniques to evaluate application and system softwareperformance for wide-area distributed systems. There are three key components to this proposal:1. To explore a range of compiler-supported strategies for highly efficient simulation of dynamic, large-scale systems and applications.2. To use these strategies in developing a practical performance tool for wide-area distributed systems, byextending our existing compiler and simulation infrastructure. This requires addressing additionalchallenges raised by the dynamic nature of these systems and the lack of well-defined metrics tomeasure effective application level performance in such environments.3. To evaluate the effectiveness of these strategies for real world distributed applications such as SF-Express, a large-scale distributed interactive simulation (DIS) environment, and a distributed video-on-demand server (a distributed multimedia application).The proposed research program builds on a collaboration of several years between the PIs' research groups,and brings together key strengths in parallel simulation of large-scale parallel programs, parallel simulation of wide-area networks, and in parallelizing compilers and their use for supporting performance evaluation.This program of research also complements the ongoing software efforts for wide-area systems that areaimed at developing operating system services (e.g., Globus, Legion, and WebOS) and programmingenvironments (e.g., Legion, Globe, and GrADS). As such, the proposed research program represents anessential third leg of software support for the development and deployment of successful wide-areadistributed systems.