GDBase: An Engine for Scalable offline Debugging

在领导类系统的运行中，大规模的调试是一个持续的挑战。虽然在研究和商业领域中对并行调试给予了大量的关注，但大多数并行调试器都依赖于视觉主题来传递有关程序执行和通信的信息。这些方法对于理解和调试数以万计的任务中的程序功能来说是无价的，但是对于发现数以万计的任务中出现的难以捉摸的问题来说就不合适了。即使接口可以在这个规模上使用（并且可以获得许可），许多商业调试器的当前迭代也无法在超过2,048个内核的情况下正常运行。大型系统的正常运行模式加剧了大规模调试的挑战。这些系统倾向于以批处理模式24x7运行，旨在最大限度地提高系统利用率和吞吐量。在交互模式中协调大量的时间来操作系统的大部分是很困难的，最坏的情况是成本过高。在这个项目中，研究人员将对GDBase调试框架进行设计。GDBase旨在解决在大型系统上进行调试的三个关键问题。缺乏可伸缩性——大多数商业调试器都过于重量级，无法在数千个内核上启动或提供可用的接口。GDBase已经测试了12k核。在我们的测试中，商业滴滴涕调试器在大约2k后失败。使用成本高——GDBase通过生产批处理系统运行作业，不需要长时间占用机器的大部分时间进行交互式调试运行，也不需要通过固定的时间保留来降低利用率，从而阻止其他作业的运行。信息过载——所有调试信息都存储在关系数据库中，以后可以由许多分析工具挖掘，而不是无法扩展的图形界面或千兆字节的printf文本。这个建议的智力价值在于采用了一种新颖的方法来处理可伸缩的调试。GDBase集成了一个轻量级的离线架构，用于性能分析工具的数据收集，一个数据库存储库用于大量调试信息和交叉运行分析，以及一个API用于创建新的代理，以提供各种接口和分析工具。该提议的更广泛影响是能够提高为大规模网络基础设施开发应用程序的所有领域的生产力。GDBase还将使大规模调试更加广泛，并将纳入一系列培训和教育活动。

Debugging at large scale is a constant challenge in the operation of leadership class systems. While there has been substantial attention paid to parallel debugging in general in both the research and commercial arenas, most parallel debuggers rely on a visual motif for conveying information about program execution and communication. These approaches tend to be invaluable for understanding and debugging program function at tens of tasks, but are simply not suitable for finding elusive problems that occur at tens of thousands of tasks. Even if the interfaces were useable at that scale (and licenses were available), current iterations of many commercial debuggers fail to function properly beyond 2,048 cores. The challenge of debugging at large scale is exacerbated by the normal mode of operation of very large systems. These systems tend to run 24x7 in a batch mode intended to maximize system utilization and throughput. Negotiating the large blocks of time to operate a substantial fraction of the system in an interactive mode is difficult at best, and cost-prohibitive at worst.In this project investigators will lay out the design for the GDBase debugging framework. GDBase is designed to attack three of the key problems of debugging on the largest systems.1.) Lack of Scalability - most commercial debuggers are too heavyweight to launch or provide a useable interface at many thousands of cores. GDBase has been tested to 12k cores. In our testing, the commercial DDT debugger fails after about 2k.2.) High Cost of use - GDBase runs jobs through the production batch systems with no need for long periods tying up large fractions of the machine for interactive debugging runs and no need to lower utilization through fixed time reservations that prevent other jobs from running.3.) Information overload - Rather than a graphical interface that fails to scale, or gigabytes of printf text, all debugging information is stored in a relational database that can be mined later by a number of analysis tools.The intellectual merit of this proposal lies in the novel approach taken to tackle scalable debugging. GDBase incorporates a lightweight, offline architecture for data collection inspired by performance analysis tools, a database repository for vast quantities of debugging information and cross-run analysis, and an API to create new agents for providing a variety of interface and analysis tools. The broader impacts of this proposal are the ability to lift productivity across all domains that develop applications for large scale cyberinfrastructure. GDBase will also make large scale debugging more widely available, and will be incorporated in a range of training and education activities.