Collaborative Research: SHF: Medium: Causal Performance Debugging for Highly-Configurable Systems

合作研究：SHF：中：高度可配置系统的因果性能调试

• 批准号: 2107463
• 负责人: Pooyan Jamshidi
• 金额: $ 40.7万
• 依托单位: University of South Carolina at Columbia
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2107463&HistoricalAwards=false
• 关键词: Collaborative; Research; SHF; Medium; Causal

Software performance is critical for most software systems to achieve scale and limit operating costs and energy consumption. As modern software systems, such as big data and machine-learning systems, are increasingly built by composing many reusable infrastructure components and deployed on distributed and heterogeneous hardware, developers have powerful tools and abstractions at their fingertips, and as a result face immense configuration complexity. Software and hardware need to be selected and configured carefully to achieve high performance for a given system and task. Unfortunately, in practice, performance faults and misconfigurations are common, where a system performs much worse than expected, not achieving its mission or simply wasting cost and energy. In large configuration spaces, end-users and developers face severe challenges in understanding and fixing performance faults by changing software configuration, changing hardware deployment, or modifying the software's code itself. Current approaches that model system performance by analyzing correlations among performance measurements and options are slow and may produce misleading results, obfuscating the actual causes of performance faults. Even if they can fix the problem, most of them cannot explain why (1) the obtained configurations are the real cause of the problem, and (2) a user/developer should consider the proposed recommendations. In both cases, the lack of explainability is a big issue. The project is intended to initiate a paradigm shift in today's testing and debugging methodology for complex, highly configurable systems, thereby positively impacting a broad range of industrial sectors relying on complex, highly configurable systems. Specifically, the project contributes to substantial energy savings and reduced carbon emissions, especially for the many big-data and machine-learning systems that operate at a massive scale. Finally, the research is providing valuable training for involved students from diverse backgrounds in research and generating high-quality researchers and practitioners for society. This project develops and evaluates foundations and tools for a causal approach to performance modeling and performance debugging. This project introduces the new concept of causal performance models that are learned using causal structure learning by intervening over configuration options and observing system performance regarding (multiple) performance objectives, rather than just analyzing correlations. Causal models enable causal inference and counterfactual reasoning for numerous tasks, including debugging performance faults and misconfigurations. Based on a solid technical foundation of causal modeling and extensive experience with performance modeling for configurable systems, this project develops innovations in three thrusts: (1) It designs and refines a causal modeling approach for software performance of systems composed of multiple configurable components, using innovations in sampling strategies, code analysis, compositional reasoning, and transfer learning to build accurate causal models efficiently. (2) It develops and evaluates user-facing tool support, based on causal models, to help users select well-performing configurations for their specific tasks and hardware and resolve misconfiguration faults with configuration changes, highlighting the (causal) performance impact of configuration decisions and providing a Pareto analysis of involved tradeoffs. (3) It develops and evaluates developer-facing tool support to foster code-level debugging and documentation. Finally, all contributions are being evaluated end-to-end with developers on real performance faults, showing how both users and developers benefit from causal models and related tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

软件性能对于大多数软件系统实现规模并限制运营成本和能源消耗至关重要。随着大数据和机器学习系统等现代软件系统越来越多地通过组成许多可重用的基础设施组件来构建，并部署在分布式和异类硬件上，开发人员拥有强大的工具和指尖上的抽象，因此面临着巨大的配置复杂性。需要仔细选择和配置软件和硬件，才能为给定的系统和任务实现高性能。不幸的是，在实践中，性能故障和错误配置是常见的，系统的性能比预期差得多，无法完成其任务，或者只是浪费成本和能源。在大型配置空间中，终端用户和开发人员面临着通过更改软件配置、更改硬件部署或修改软件代码本身来了解和修复性能故障的严峻挑战。当前通过分析性能度量和选项之间的相关性来建模系统性能的方法速度很慢，可能会产生误导性的结果，混淆了性能故障的实际原因。即使他们可以解决问题，他们中的大多数人也无法解释为什么(1)获得的配置是问题的真正原因，以及(2)用户/开发人员应该考虑建议的建议。在这两种情况下，缺乏可解释性都是一个大问题。该项目旨在启动当今复杂、高度可配置系统测试和调试方法的范式转变，从而对依赖复杂、高度可配置系统的广泛工业部门产生积极影响。具体地说，该项目有助于大幅节省能源和减少碳排放，特别是对于许多大规模运行的大数据和机器学习系统。最后，这项研究为来自不同背景的学生提供了宝贵的研究培训，并为社会培养高素质的研究人员和实践者。该项目开发和评估用于性能建模和性能调试的因果方法的基础和工具。该项目引入了因果性能模型的新概念，通过介入配置选项并观察有关(多个)性能目标的系统性能，而不仅仅是分析相关性，使用因果结构学习来学习因果性能模型。因果模型允许对许多任务进行因果推理和反事实推理，包括调试性能故障和错误配置。该项目基于扎实的因果建模技术基础和丰富的可配置系统性能建模经验，在以下三个方面进行了创新：(1)设计并提炼了一种面向多个可配置组件组成的系统软件性能的因果建模方法，在抽样策略、代码分析、组合推理、迁移学习等方面进行了创新，有效地构建了准确的因果模型。(2)它基于因果模型开发和评估面向用户的工具支持，以帮助用户为其特定任务和硬件选择性能良好的配置，并通过配置更改解决错误配置故障，突出配置决策的(因果)性能影响，并提供涉及的权衡的帕累托分析。(3)开发和评估面向开发人员的工具支持，以促进代码级调试和文档编制。最后，所有的贡献都是与开发人员就真正的性能缺陷进行端到端的评估，显示用户和开发人员如何从因果模型和相关工具中受益。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization Approach for Deep Neural Networks

• DOI: 10.1613/jair.1.14139
• 发表时间: 2020-01
• 期刊: J. Artif. Intell. Res.
• 作者: Md Shahriar Iqbal;Jianhai Su;Lars Kotthoff;Pooyan Jamshidi

Getting the Best Bang For Your Buck: Choosing What to Evaluate for Faster Bayesian Optimization

物有所值：选择评估内容以加快贝叶斯优化速度

• 发表时间: 2022
• 期刊: PMLR 188:6
• 作者: Iqbal, Md Shahriar;Su, Jianhai;Kotthoff, Lars;Jamshidi, Pooyan
• 通讯作者: Jamshidi, Pooyan

Unicorn: reasoning about configurable system performance through the lens of causality

• DOI: 10.1145/3492321.3519575
• 发表时间: 2022-01
• 期刊: Proceedings of the Seventeenth European Conference on Computer Systems
• 作者: Md Shahriar Iqbal;R. Krishna;Mohammad Ali Javidian;Baishakhi Ray;Pooyan Jamshidi

On Debugging the Performance of Configurable Software Systems: Developer Needs and Tailored Tool Support

• DOI: 10.1145/3510003.3510043
• 发表时间: 2022-03
• 期刊: 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)
• 作者: Miguel Velez;Pooyan Jamshidi;Norbert Siegmund;S. Apel;Christian Kastner

FELARE: Fair Scheduling of Machine Learning Tasks on Heterogeneous Edge Systems

• DOI: 10.1109/cloud55607.2022.00069
• 发表时间: 2022-05
• 期刊: 2022 IEEE 15th International Conference on Cloud Computing (CLOUD)
• 作者: Ali Mokhtari;Pooyan Jamshidi;M. Salehi