Collaborative Research: SHF: Medium: Causal Performance Debugging for Highly-Configurable Systems

合作研究：SHF：中：高度可配置系统的因果性能调试

• 批准号: 2107405
• 负责人: Baishakhi Ray
• 金额: $ 37.3万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2107405&HistoricalAwards=false
• 关键词: Collaborative; Research; SHF; Medium; Causal

Software performance is critical for most software systems to achieve scale and limit operating costs and energy consumption. As modern software systems, such as big data and machine-learning systems, are increasingly built by composing many reusable infrastructure components and deployed on distributed and heterogeneous hardware, developers have powerful tools and abstractions at their fingertips, and as a result face immense configuration complexity. Software and hardware need to be selected and configured carefully to achieve high performance for a given system and task. Unfortunately, in practice, performance faults and misconfigurations are common, where a system performs much worse than expected, not achieving its mission or simply wasting cost and energy. In large configuration spaces, end-users and developers face severe challenges in understanding and fixing performance faults by changing software configuration, changing hardware deployment, or modifying the software's code itself. Current approaches that model system performance by analyzing correlations among performance measurements and options are slow and may produce misleading results, obfuscating the actual causes of performance faults. Even if they can fix the problem, most of them cannot explain why (1) the obtained configurations are the real cause of the problem, and (2) a user/developer should consider the proposed recommendations. In both cases, the lack of explainability is a big issue. The project is intended to initiate a paradigm shift in today's testing and debugging methodology for complex, highly configurable systems, thereby positively impacting a broad range of industrial sectors relying on complex, highly configurable systems. Specifically, the project contributes to substantial energy savings and reduced carbon emissions, especially for the many big-data and machine-learning systems that operate at a massive scale. Finally, the research is providing valuable training for involved students from diverse backgrounds in research and generating high-quality researchers and practitioners for society. This project develops and evaluates foundations and tools for a causal approach to performance modeling and performance debugging. This project introduces the new concept of causal performance models that are learned using causal structure learning by intervening over configuration options and observing system performance regarding (multiple) performance objectives, rather than just analyzing correlations. Causal models enable causal inference and counterfactual reasoning for numerous tasks, including debugging performance faults and misconfigurations. Based on a solid technical foundation of causal modeling and extensive experience with performance modeling for configurable systems, this project develops innovations in three thrusts: (1) It designs and refines a causal modeling approach for software performance of systems composed of multiple configurable components, using innovations in sampling strategies, code analysis, compositional reasoning, and transfer learning to build accurate causal models efficiently. (2) It develops and evaluates user-facing tool support, based on causal models, to help users select well-performing configurations for their specific tasks and hardware and resolve misconfiguration faults with configuration changes, highlighting the (causal) performance impact of configuration decisions and providing a Pareto analysis of involved tradeoffs. (3) It develops and evaluates developer-facing tool support to foster code-level debugging and documentation. Finally, all contributions are being evaluated end-to-end with developers on real performance faults, showing how both users and developers benefit from causal models and related tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

软件性能对于大多数软件系统实现规模和限制运营成本和能耗至关重要。随着现代软件系统（如大数据和机器学习系统）越来越多地通过组成许多可重用的基础设施组件并部署在分布式和异构硬件上来构建，开发人员拥有强大的工具和抽象，因此面临巨大的配置复杂性。需要仔细选择和配置软件和硬件，以实现给定系统和任务的高性能。不幸的是，在实践中，性能故障和错误配置是常见的，其中系统的性能比预期差得多，没有实现其使命或只是浪费成本和能量。在大型配置空间中，最终用户和开发人员在通过更改软件配置、更改硬件部署或修改软件代码本身来理解和修复性能故障方面面临严峻的挑战。目前的方法，通过分析性能测量和选项之间的相关性来建模系统性能是缓慢的，可能会产生误导性的结果，混淆性能故障的实际原因。即使他们可以解决问题，他们中的大多数人也无法解释为什么（1）获得的配置是问题的真实的原因，以及（2）用户/开发人员应该考虑提出的建议。在这两种情况下，缺乏可解释性是一个大问题。该项目旨在为当今复杂、高度可配置的系统的测试和调试方法带来范式转变，从而对依赖复杂、高度可配置系统的广泛工业部门产生积极影响。具体而言，该项目有助于节省大量能源和减少碳排放，特别是对于许多大规模运行的大数据和机器学习系统。最后，这项研究为来自不同背景的学生提供了宝贵的培训，并为社会培养了高素质的研究人员和从业人员。这个项目开发和评估的基础和工具的因果关系的方法，性能建模和性能调试。该项目引入了因果性能模型的新概念，通过干预配置选项和观察系统性能（多个）性能目标，而不仅仅是分析相关性，使用因果结构学习来学习。因果模型使因果推理和反事实推理的许多任务，包括调试性能故障和错误配置。基于坚实的因果建模技术基础和丰富的可配置系统性能建模经验，本项目在三个方面进行了创新：（1）设计和改进了一种用于多个可配置组件组成的系统的软件性能的因果建模方法，使用采样策略，代码分析，组合推理和迁移学习的创新来有效地建立准确的因果模型。(2)它基于因果模型开发和评估面向用户的工具支持，以帮助用户为其特定任务和硬件选择性能良好的配置，并通过配置更改解决错误配置故障，突出配置决策的（因果）性能影响，并提供对所涉及权衡的帕累托分析。(3)它开发和评估面向开发人员的工具支持，以促进代码级调试和文档编制。最后，所有的贡献都将在真实的性能故障上与开发人员进行端到端的评估，展示用户和开发人员如何从因果模型和相关工具中受益。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。