A Comprehensive Framework for the Automatic Evaluation of the Quality of ML-based Software Systems

如今，机器学习软件系统（MLSS）已经成为我们日常生活的一部分（例如，推荐系统，语音识别，人脸检测）。越来越多的公司需要使用机器学习来解决他们的业务问题。MLSS的核心是ML模型。这些模型作为软件实现，并且像任何其他软件一样，质量保证是必要的。mlss的质量评价是一项具有挑战性的任务，也是目前文献研究的热点。随着mlss部署的不断增加，对其服务质量的要求也越来越高。此类系统的错误或错误决策可能导致其他系统故障，造成重大经济损失，甚至威胁到人类生命。在本项目中，为了对mlss中的ML模型进行全面的质量评估，将考虑ML模型在其生命周期的不同阶段作为系统一部分的作用。将评估机器学习模型质量的各个方面，从性能和鲁棒性（如预测准确性、数据偏差和方差）到可扩展性、硬件/软件需求、复杂性、用户接受度和可解释性。将实现一个多目标框架，以考虑mlss中ML模型质量的所有属性。最后，我们将把我们提出的解决方案聚合在一个实用的工具集中，以自动评估、验证和跟踪机器学习模型在整个生命周期中的质量。该工具集将被集成到最先进的工具中，用于软件系统的持续集成和交付。该工具集将为MoovAI以及其他魁北克和加拿大公司提供使用ML的工具，在蓬勃发展的ML和AI市场中具有竞争优势。他们的MLSS的可靠性将是赢得新市场的重要资产。使用高质量的机器学习模型和可靠的MLSS将增加魁北克省和加拿大工业界对机器学习/人工智能技术的信任。

Nowadays, Machine Learning Software Systems (MLSS)s have become a part of our daily life (e.g., recommendation systems, speech recognition, face detection). An increasing demand is observed in various companies to employ ML for solving problems in their business. The heart of MLSS is an ML model. These models are implemented as software and like any other software, quality assurance is necessary. The quality assessment of MLSSs is regarded as a challenging task and is currently a hot research topic in the literature. According to the growing deployment of MLSSs, there is a strong need for ensuring their serving quality. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threat to human life.In this project, for a comprehensive quality assessment of ML models in MLSSs, the role of the ML model as a part of the system at different stages of its life cycle will be considered. Various aspects of the quality of ML models from performance and robustness (like prediction accuracy, data bias, and variance), to scalability, hardware/software demand, complexity, user acceptance, and explainability will be evaluated. A multi-objective framework will be implemented to take into account all properties of ML model quality in MLSSs. Finally, we will aggregate our proposed solutions in a practical toolset to automatically evaluate, validate, and track the quality of ML models throughout their life cycle. The toolset will be integrated into the state-of-the-art tools for continuous integration and delivery of software systems. This toolset will provide MoovAI as well as other Quebec and Candian companies using ML, with a competitive edge in the booming ML and AI market. The dependability of their MLSS will be a great asset in winning new markets. The usage of high-quality ML models and reliable MLSS will increase trust in ML/AI technologies across the Quebec and Canadian industry.