A Comprehensive Framework for the Automatic Evaluation of the Quality of ML-based Software Systems

基于机器学习的软件系统质量自动评估的综合框架

• 批准号: 561420-2020
• 负责人: Khomh, FoutseFTSE
• 金额: $ 6.66万
• 依托单位: École Polytechnique de Montréal
• 依托单位国家: 加拿大
• 项目类别: Alliance Grants
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=754164
• 关键词: Comprehensive; Framework; Automatic; Evaluation; Quality

Nowadays, Machine Learning Software Systems (MLSS)s have become a part of our daily life (e.g., recommendation systems, speech recognition, face detection). An increasing demand is observed in various companies to employ ML for solving problems in their business. The heart of MLSS is an ML model. These models are implemented as software and like any other software, quality assurance is necessary. The quality assessment of MLSSs is regarded as a challenging task and is currently a hot research topic in the literature. According to the growing deployment of MLSSs, there is a strong need for ensuring their serving quality. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threat to human life.In this project, for a comprehensive quality assessment of ML models in MLSSs, the role of the ML model as a part of the system at different stages of its life cycle will be considered. Various aspects of the quality of ML models from performance and robustness (like prediction accuracy, data bias, and variance), to scalability, hardware/software demand, complexity, user acceptance, and explainability will be evaluated. A multi-objective framework will be implemented to take into account all properties of ML model quality in MLSSs. Finally, we will aggregate our proposed solutions in a practical toolset to automatically evaluate, validate, and track the quality of ML models throughout their life cycle. The toolset will be integrated into the state-of-the-art tools for continuous integration and delivery of software systems. This toolset will provide MoovAI as well as other Quebec and Candian companies using ML, with a competitive edge in the booming ML and AI market. The dependability of their MLSS will be a great asset in winning new markets. The usage of high-quality ML models and reliable MLSS will increase trust in ML/AI technologies across the Quebec and Canadian industry.

如今，机器学习软件系统（MLSS）已经成为我们日常生活的一部分（例如，推荐系统、语音识别、面部检测）。越来越多的公司需要使用ML来解决业务中的问题。MLSS的核心是ML模型。这些模型作为软件实现，与任何其他软件一样，质量保证是必要的。MLSS的质量评估被认为是一项具有挑战性的任务，是目前文献中的一个热点研究课题。随着MLSS部署的不断增加，迫切需要确保其服务质量。此类系统的错误或错误决策可能导致其他系统故障，重大经济损失，甚至威胁到人类生命。在本项目中，为了对MLSS中的ML模型进行全面的质量评估，ML模型作为系统的一部分在其生命周期的不同阶段的作用将被考虑。将评估ML模型质量的各个方面，从性能和鲁棒性（如预测准确性，数据偏差和方差）到可扩展性，硬件/软件需求，复杂性，用户接受度和可解释性。将实施一个多目标框架，以考虑MLSS中ML模型质量的所有属性。最后，我们将把我们提出的解决方案聚合到一个实用的工具集中，以在ML模型的整个生命周期中自动评估、验证和跟踪ML模型的质量。该工具集将被纳入最先进的工具，以持续集成和交付软件系统。该工具集将为MoovAI以及其他使用ML的魁北克和加拿大公司提供在蓬勃发展的ML和AI市场中的竞争优势。他们的MLSS的可靠性将是赢得新市场的巨大资产。使用高质量的ML模型和可靠的MLSS将增加魁北克和加拿大行业对ML/AI技术的信任。