Testing, Debugging and Repairing Machine Learning Software at the System Level

系统级测试、调试和修复机器学习软件

• 批准号: RGPIN-2021-02549
• 负责人: Ma, Lei
• 金额: $ 3.86万
• 依托单位: University of Alberta
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=752184
• 关键词: Testing; Debugging; Repairing; Machine; Learning

Machine learning (ML) has become the driving force of innovation in many application domains. However, current state-of-the-art ML software still suffers from quality issues. Different from traditional software, ML software adopts the data-driven programming paradigm. Instead of manually programming the decision logic (e.g., in form of source code), the decision logic of the ML software is automatically learned via training and encodes in a model (e.g., neural network). An ML model is often difficult to interpret and understand, calling for novel quality assurance (QA) methods. In practice, an ML model is often not used standalone but integrated as a component into a larger system, including both traditional software and ML models. While some recent progress is made on testing and analysis ML models, systematic research of QA for ML software at the system level is still largely untouched so far. Quality assurance at the ML system level is challenging and requires considering the components of ML models, traditional software, and their interactions. In particular, 1) The ML model behaviors are often difficult to understand. What is the role of an ML model and how its incorrect behaviors impact the whole system? 2) How to effectively detect the defects in the huge testing space of an ML software system? 3) With the error triggering tests, how to debug and identify the defect modules? 4) For system incorrect behaviors caused by ML models, how to repair them to improve system quality? This research program aims to address these challenges and propose novel methods of testing, debugging, and repairing for ML-driven software at the system level, providing key quality assurance supports to establish trustworthy intelligent software. 1) First, I plan to perform a large-scale empirical study to systematically investigate roles and defect impacts of ML models in state-of-the-art ML systems. 2) Then, I will propose an effective testing framework to detect the potential defects of ML at the system level. 3) With the found defects, I plan to design automated debugging techniques to localize the potentially incorrect modules. 4) Regarding the system defects introduced by ML models, I will further propose automated repairing methods to enhance the ML system quality. Large-scale experiments on open source and industrial ML software systems will be conducted to evaluate the advantage, practical value and limitation of proposed techniques. This outcome of this research will originally provide an initial set of key methods to detect, debug, and repair ML software at the system level, which can greatly accelerate the ML system development process with better quality assurance support, potentially impacting many industrial domains. This program will train nine highly qualified personnel (HQP) and provide them with the equity, diversity and inclusivity (EDI) platform to participate and contribute to the state-of-the-art intelligent software engineering research.

机器学习（ML）已成为许多应用领域创新的驱动力。然而，当前最先进的机器学习软件仍然存在质量问题。与传统软件不同，机器学习软件采用数据驱动的编程范式。机器学习软件的决策逻辑不是手动编程决策逻辑（例如以源代码的形式），而是通过模型（例如神经网络）中的训练和编码自动学习。机器学习模型通常难以解释和理解，需要新颖的质量保证 (QA) 方法。在实践中，机器学习模型通常不会单独使用，而是作为组件集成到更大的系统中，包括传统软件和机器学习模型。虽然最近在测试和分析机器学习模型方面取得了一些进展，但迄今为止，机器学习软件在系统级别的质量保证的系统研究仍然基本上没有触及。机器学习系统级别的质量保证具有挑战性，需要考虑机器学习模型的组件、传统软件及其交互。特别是，1）ML 模型的行为通常难以理解。机器学习模型的作用是什么？它的错误行为如何影响整个系统？ 2）如何在机器学习软件系统巨大的测试空间中有效检测缺陷？ 3）通过错误触发测试，如何调试和识别缺陷模块？ 4）对于ML模型导致的系统错误行为，如何修复以提高系统质量？该研究项目旨在应对这些挑战，并提出在系统级别对机器学习驱动的软件进行测试、调试和修复的新方法，为建立值得信赖的智能软件提供关键的质量保证支持。 1）首先，我计划进行大规模的实证研究，系统地研究 ML 模型在最先进的 ML 系统中的作用和缺陷影响。 2）然后，我将提出一个有效的测试框架来检测系统级别的ML潜在缺陷。 3）根据发现的缺陷，我计划设计自动调试技术来定位可能不正确的模块。 4）针对ML模型引入的系统缺陷，我将进一步提出自动修复方法来提高ML系统质量。 将在开源和工业机器学习软件系统上进行大规模实验，以评估所提出技术的优势、实用价值和局限性。这项研究的成果最初将提供一套初始的关键方法来在系统级别检测、调试和修复机器学习软件，这可以通过更好的质量保证支持极大地加速机器学习系统的开发过程，从而可能影响许多工业领域。该项目将培养九名高素质人才（HQP），并为他们提供公平、多样性和包容性（EDI）平台，参与最先进的智能软件工程研究并做出贡献。