CRII: SHF: Assessing and Profiling Continuous Integration for Machine Learning Applications

该奖项全部或部分由《2021年美国救援计划法案》（公法117-2）资助。持续集成（CI）是一种被广泛采用的软件开发实践，用于更快的代码更改集成和软件质量属性的维护。与此同时，机器学习（ML），包括深度学习（DL），在解决复杂问题方面正在迅速普及。像典型的软件一样，机器学习应用程序也需要多次迭代来提高软件质量。然而，迭代的机器学习应用程序开发过程在采用CI方面面临着三个方面的更高层次的困难。首先，开发人员对CI工作流中管理ML数据、模型和代码缺乏系统的理解。对于基于ml的系统，定义CI工作流的过程目前在本质上更具实验性。其次，现有CI系统在处理以ML为中心的挑战方面存在不足，例如定义ML模型的评估条件、制定复杂的构建步骤、长构建和集成时间等。对于ML应用程序，由于数据、模型、代码等的复杂依赖关系，构建过程要复杂得多。第三，由于数据、模型、代码等之间的复杂互连以及ML应用程序不断变化的性质，数据科学家缺乏采用和维护CI配置的技术支持。对于那些对CI了解有限或一无所知的数据科学家来说，为ML应用程序采用CI变得越来越困难，即使采用了CI，也需要太多的手工工作。该项目将取得进展，以获取有关当前采用CI进行机器学习应用的可行性和有效性的知识。这将有助于理解ML应用程序的CI工作流程，并确定改进范围。此外，该项目将开发一个新的CI分析框架，以在ML应用程序的异构工件（例如，数据、模型、代码等）之间生成依赖关系图。这些知识和框架将作为未来研究ML CI配置的自动生成和维护、挖掘软件存储库、构建ML CI优化和监控系统的基础。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Continuous integration (CI) is a widely adopted software development practice for faster code change integration and maintenance of software quality attributes. At the same time, machine learning (ML), including deep learning (DL), is quickly gaining popularity for solving complex problems. Like typical software, ML applications also require many iterations to improve software quality. However, iterative ML application-development processes face higher-level difficulties in adopting CI in three aspects. First, developers lack systematic understanding for managing ML data, models and code in the CI workflow. For ML-based systems, the process to define the CI workflow is currently much more experimental in nature. Second, existing CI systems are lacking in the handling of ML-centric challenges such as defining evaluation conditions of ML models, formulating complicated build steps, long build and integration time, etc. For ML applications, the build process is much more complicated due to the complex dependency of data, model, code, etc. Third, data scientists lack the technical support to adopt and maintain CI configurations due to complex interconnections among data, model, code, etc. and the changing nature of the ML applications. For data scientists with limited or no knowledge of CI, it becomes increasingly difficult to adopt CI for ML applications, and even if adopted it requires too much manual effort. The project will make progress in acquiring knowledge on the feasibility and effectiveness of the current adoption of CI for ML applications. This will assist in understanding the workflow of CI for ML applications and identifying improvement scopes. Moreover, the project will develop a novel CI profiling framework to generate a dependency graph among heterogeneous artifacts (e.g., data, model, code, etc.) of ML applications. The knowledge and framework will serve as the basis for future research on automatic generation and maintenance of ML CI configuration, mining software repositories, build optimization and monitoring systems for ML CI.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.