A Comprehensive Framework for the Automatic Evaluation of the Quality of ML-based Software Systems

如今，机器学习软件系统(MLSS)S已经成为我们日常生活(如推荐系统、语音识别、人脸检测)的一部分。人们发现，越来越多的公司使用ML来解决业务中的问题。MLSS的核心是ML模型。这些模型是以软件的形式实现的，与任何其他软件一样，质量保证是必要的。MLSS的质量评估被认为是一项具有挑战性的任务，也是目前文献中的一个热门研究课题。随着MLSS的部署日益增多，迫切需要确保其服务质量。在这个项目中，对于MLSS中ML模型的综合质量评估，将考虑ML模型在系统生命周期的不同阶段所扮演的角色。将评估ML模型质量的各个方面，从性能和稳健性(如预测精度、数据偏差和方差)，到可伸缩性、硬件/软件需求、复杂性、用户接受度和可解释性。将实施一个多目标框架，以考虑MLSS中ML模型质量的所有属性。最后，我们将把我们提出的解决方案聚合到一个实用的工具集中，以在ML模型的整个生命周期中自动评估、验证和跟踪它们的质量。该工具包将被整合到最先进的工具中，以实现软件系统的持续集成和交付。该工具集将为MoovAI以及其他使用ML的魁北克和加拿大公司提供在蓬勃发展的ML和AI市场上的竞争优势。它们MLS的可靠性将是赢得新市场的一大资产。使用高质量的ML模型和可靠的MLSS将增加魁北克和加拿大整个行业对ML/AI技术的信任。

Nowadays, Machine Learning Software Systems (MLSS)s have become a part of our daily life (e.g., recommendation systems, speech recognition, face detection). An increasing demand is observed in various companies to employ ML for solving problems in their business. The heart of MLSS is an ML model. These models are implemented as software and like any other software, quality assurance is necessary. The quality assessment of MLSSs is regarded as a challenging task and is currently a hot research topic in the literature. According to the growing deployment of MLSSs, there is a strong need for ensuring their serving quality. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threat to human life.In this project, for a comprehensive quality assessment of ML models in MLSSs, the role of the ML model as a part of the system at different stages of its life cycle will be considered. Various aspects of the quality of ML models from performance and robustness (like prediction accuracy, data bias, and variance), to scalability, hardware/software demand, complexity, user acceptance, and explainability will be evaluated. A multi-objective framework will be implemented to take into account all properties of ML model quality in MLSSs. Finally, we will aggregate our proposed solutions in a practical toolset to automatically evaluate, validate, and track the quality of ML models throughout their life cycle. The toolset will be integrated into the state-of-the-art tools for continuous integration and delivery of software systems. This toolset will provide MoovAI as well as other Quebec and Candian companies using ML, with a competitive edge in the booming ML and AI market. The dependability of their MLSS will be a great asset in winning new markets. The usage of high-quality ML models and reliable MLSS will increase trust in ML/AI technologies across the Quebec and Canadian industry.