GDBase: An Engine for Scalable offline Debugging

大规模调试是领导力级别系统运行中的一个持续挑战。虽然在研究和商业领域普遍对并行调试有很大的关注，但大多数并行调试器依赖于一个可视化的主题来传递关于程序执行和通信的信息。这些方法对于理解和调试在数十个任务中的程序功能往往是非常有价值的，但根本不适合于发现在数万个任务中出现的难以捉摸的问题。即使接口在这种规模下可用(并且许可证可用)，许多商业调试器当前的迭代也无法在超过2,048个内核的情况下正常运行。超大型系统的正常运行模式加剧了大规模调试的挑战。这些系统倾向于以批处理模式全天候运行，旨在最大化系统利用率和吞吐量。在这个项目中，调查人员将为GDBase调试框架设计出GDBase调试框架。GDBase旨在解决在最大系统上调试的三个关键问题。缺乏可伸缩性-大多数商业调试器太重，无法在数千个内核上推出或提供可用的接口。GDBase已经过12000核的测试。在我们的测试中，商业DDT调试器在大约2k之后失败。)使用成本高-GDBase在生产批处理系统中运行作业，不需要长时间占用机器的很大一部分用于交互调试运行，也不需要通过阻止其他作业运行的固定时间预留来降低利用率。信息过载-所有调试信息都存储在关系数据库中，而不是无法扩展的图形界面或千兆字节的打印文本。这一建议的智力优势在于采用了新的方法来解决可伸缩调试问题。GDBase结合了受性能分析工具启发的用于数据收集的轻量级离线体系结构、用于存储大量调试信息和交叉运行分析的数据库存储库，以及用于创建用于提供各种接口和分析工具的新代理的API。这一提议的更广泛影响是，能够提高为大规模网络基础设施开发应用程序的所有领域的生产率。GDBase还将使大规模调试更加广泛，并将被纳入一系列培训和教育活动中。

Debugging at large scale is a constant challenge in the operation of leadership class systems. While there has been substantial attention paid to parallel debugging in general in both the research and commercial arenas, most parallel debuggers rely on a visual motif for conveying information about program execution and communication. These approaches tend to be invaluable for understanding and debugging program function at tens of tasks, but are simply not suitable for finding elusive problems that occur at tens of thousands of tasks. Even if the interfaces were useable at that scale (and licenses were available), current iterations of many commercial debuggers fail to function properly beyond 2,048 cores. The challenge of debugging at large scale is exacerbated by the normal mode of operation of very large systems. These systems tend to run 24x7 in a batch mode intended to maximize system utilization and throughput. Negotiating the large blocks of time to operate a substantial fraction of the system in an interactive mode is difficult at best, and cost-prohibitive at worst.In this project investigators will lay out the design for the GDBase debugging framework. GDBase is designed to attack three of the key problems of debugging on the largest systems.1.) Lack of Scalability - most commercial debuggers are too heavyweight to launch or provide a useable interface at many thousands of cores. GDBase has been tested to 12k cores. In our testing, the commercial DDT debugger fails after about 2k.2.) High Cost of use - GDBase runs jobs through the production batch systems with no need for long periods tying up large fractions of the machine for interactive debugging runs and no need to lower utilization through fixed time reservations that prevent other jobs from running.3.) Information overload - Rather than a graphical interface that fails to scale, or gigabytes of printf text, all debugging information is stored in a relational database that can be mined later by a number of analysis tools.The intellectual merit of this proposal lies in the novel approach taken to tackle scalable debugging. GDBase incorporates a lightweight, offline architecture for data collection inspired by performance analysis tools, a database repository for vast quantities of debugging information and cross-run analysis, and an API to create new agents for providing a variety of interface and analysis tools. The broader impacts of this proposal are the ability to lift productivity across all domains that develop applications for large scale cyberinfrastructure. GDBase will also make large scale debugging more widely available, and will be incorporated in a range of training and education activities.