Collaborative Research: Compiler-Supported Simulation of Scalable Applications for Wide-Area Distributed Computing Systems

广域分布式计算系统代表了科学和商业计算的未来，它将带来广泛的未来应用，如分布式多媒体服务、基于Web的协作、分布式超级计算和远程沉浸等，具有巨大的经济和社会影响力。然而，在实现这一愿景之前，还需要对软件系统设计、实现和评估的几乎所有方面进行广泛的研究。离散事件模拟已经成为传统计算系统和应用程序设计和分析的重要工具。然而，由于必须模拟的软件、硬件和网络子系统的规模，在广域分布式系统上执行的应用程序的真实仿真是一项具有挑战性的任务。然而，这些组件的复杂性和复杂的，闭环的相互作用，要求组件和它们的相互作用进行建模，在足够的细节，以适当地预测其对整体系统性能的影响。在最近的工作中，PI合作获得了一些令人兴奋的，但初步的结果表明，特定的编译器信息可以大大提高效率和可扩展性的消息传递程序的模拟。例如，它表明，模拟一个可扩展的ASCI内核基准测试应用程序称为SWEEP 3D执行多达128个处理器可以模拟速度比实时通过使用并行模拟与类型的编译器优化，这是本提案的主题。此外，编译器优化的模拟可以在数千个处理器上评估非常大的数据集：它可以在多达10，000个处理器上模拟4000万个问题大小的Sweep 3D的性能。有潜在的一些其他的策略，以显着提高simulationscalability和性能，通过使用编译器的信息，没有一个已经研究到目前为止。一个全面的研究计划需要开发他们的潜力，并评估其对模拟真实的世界applications.The当前建议的重点是开发基于编译器的技术，以提高并行离散事件模拟的效率，并使用这些技术来评估应用程序和广域分布式系统softwareperformance的影响。该提案有三个关键组成部分：1。探索一系列编译器支持的策略，以实现动态、大规模系统和应用程序的高效仿真。利用这些策略，通过扩展我们现有的编译器和仿真基础设施，开发一个实用的广域分布式系统性能工具。这需要解决这些系统的动态特性所带来的额外挑战，以及缺乏定义良好的指标来衡量此类环境中有效的应用程序级性能.为了评估这些策略对于真实的世界分布式应用的有效性，例如SF-Express、大规模分布式交互仿真（DIS）环境和分布式视频点播服务器（分布式多媒体应用）。拟议的研究计划建立在PI研究小组之间多年的合作基础上，并汇集了大规模并行程序并行模拟的关键优势，广域网的并行模拟，并行编译器及其用于支持性能评估。该研究计划还补充了正在进行的广域系统软件工作，旨在开发操作系统服务（例如，Globus、Legion和WebOS）和编程工具（例如，Legion、地球仪和GrADS）。因此，拟议的研究计划代表了成功的广域分布式系统的开发和部署的软件支持的一个重要的第三站。

Wide-area distributed computing systems represent the future of scientific and commercial computing.Such systems will enable a wide range of futuristic applications with the potential for enormous economicand social impact applications such as distributed multimedia services, Web-based collaboration,distributed supercomputing, and teleimmersion. Before this vision can be realized, however, extensiveresearch will be required in virtually all aspects of software system design, implementation, and evaluation.Discrete-event simulation has been an essential tool for the design and analysis of traditional computationalsystems and applications. Realistic simulation of applications executing on wide-area distributed systems,however, is a challenging task because of the scale of the software, hardware and network subsystems thatmust be simulated. Nevertheless, the intricacy of these components and their complex, closed-loopinteractions require the components and their interactions to be modeled in sufficient detail to appropriatelypredict their impact on overall system performance.In recent work, the PIs have collaboratively obtained some exciting but preliminary results showing thatspecific compiler information can greatly enhance the efficiency and scalability of simulation of message-passing programs. For instance, it was shown that simulation of a scalable ASCI kernel benchmarkapplication called SWEEP3D executing on up to 128 processors could be simulated faster than real-time byusing parallel simulations together with the type of compiler optimizations that are the subject of thisproposal. Also, the compiler-optimized simulation can evaluate very large data sets on thousands ofprocessors: it was possible to simulate the performance of a 40 million-problem size Sweep3D for up to10,000 processors. There are potentially a number of other strategies to dramatically improve simulationscalability and performance by using compiler information, none of which have been studied so far. Acomprehensive program of research is required to develop their potential and evaluate their impact onsimulation of real world applications.The focus of the current proposal is to develop compiler-based techniques for improving the efficiency ofparallel discrete event simulation, and to use these techniques to evaluate application and system softwareperformance for wide-area distributed systems. There are three key components to this proposal:1. To explore a range of compiler-supported strategies for highly efficient simulation of dynamic, large-scale systems and applications.2. To use these strategies in developing a practical performance tool for wide-area distributed systems, byextending our existing compiler and simulation infrastructure. This requires addressing additionalchallenges raised by the dynamic nature of these systems and the lack of well-defined metrics tomeasure effective application level performance in such environments.3. To evaluate the effectiveness of these strategies for real world distributed applications such as SF-Express, a large-scale distributed interactive simulation (DIS) environment, and a distributed video-on-demand server (a distributed multimedia application).The proposed research program builds on a collaboration of several years between the PIs' research groups,and brings together key strengths in parallel simulation of large-scale parallel programs, parallel simulation of wide-area networks, and in parallelizing compilers and their use for supporting performance evaluation.This program of research also complements the ongoing software efforts for wide-area systems that areaimed at developing operating system services (e.g., Globus, Legion, and WebOS) and programmingenvironments (e.g., Legion, Globe, and GrADS). As such, the proposed research program represents anessential third leg of software support for the development and deployment of successful wide-areadistributed systems.