CSR: Medium: Improving the Interface between Machine Learning and Software Systems

机器学习即服务（MLaaS）是一种新的软件范式，它使开发人员能够访问在大规模数据集上训练的强大机器学习模型，而不需要这些开发人员知道如何训练模型或访问所需的训练数据。这种方法使机器学习可以用于更广泛的软件系统，但它也带来了新的挑战。具体而言，在提供非常通用的MLaaS接口（使其尽可能广泛地适用）的愿望与使用MLaaS的单个应用程序的特定需求之间存在紧张关系。例如，许多MLaaS提供商提供通用对象检测模型作为服务，其可以识别图片中的数万个不同对象，但大多数应用仅需要该能力的一小部分;例如，与交通有关的应用只关心可能出现在道路上的对象。本项目将探讨这种紧张关系-保持MLaaS的通用性，同时提高使用这些服务的单个应用程序的鲁棒性，准确性和性能。具体来说，该项目将首先创建一个真实世界应用程序的基准套件，以推动对由于一般MLaaS接口和特定应用程序需求之间的紧张关系而出现的软件错误的实证研究。基于这项研究，该项目将创建一组工具，自动调整软件，以解决由于在特定应用程序环境中使用通用MLaaS接口而产生的不一致和模糊性。最后，该项目将创建重构软件的方法和工具，以使用其他信息（包括MLaaS对其结果的信心），这些信息可以从MLaaS提供商获得，但通常被软件应用程序忽略。机器学习现在是影响我们日常生活的软件系统的重要组成部分，包括交通，医疗系统，甚至新闻发布。MLaaS的兴起使非专家更容易将机器学习纳入这些软件系统，但它也增加了新一类软件错误和软件故障的机会。该项目将识别和分类在大型软件系统中使用MLaaS时可能出现的新型错误，并创建工具和方法来识别和修复这些错误。通过该项目开发的所有基准、数据和软件工具将作为开源发布，以便更大的社区可以免费受益于这项工作。该项目通过提高使用机器学习服务的软件系统的正确性和性能，不仅使软件的开发变得更加容易，而且使人们使用的软件更加可靠，从而提高人们的日常生活质量。该奖项反映了NSF的法定使命，通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine-Learning-as-a-Service (MLaaS) is a new software paradigm that gives developers access to powerful machine learning models trained on massive data sets without requiring those developers know how to train the models or have access to the required training data. This approach makes machine learning accessible to a much wider range of software systems, but it also creates new challenges. Specifically, there is a tension between the desire to provide a very general MLaaS interface (to make it as widely applicable as possible) and the specific needs of individual applications that use MLaaS. For example, many MLaaS providers offer general object detection models as a service, which can recognize tens of thousands of different objects in a picture, but most applications require only a small subset of that capability; e.g., applications concerned with traffic only care about objects that could appear on a roadway. This project will explore this tension – to preserve the generality of MLaaS while improving the robustness, accuracy, and performance of individual applications that use these services. Specifically, the project will first create a benchmark suite of real-world applications to drive an empirical study of the software bugs that arise due to the tension between general MLaaS interfaces and specific application needs. Based on that study, the project will create a set of tools that automatically adapt software to fix inconsistencies and ambiguities that arise due to the use of general MLaaS interfaces in application-specific contexts. Finally, the project will create methods and tools for refactoring software to use additional information –including the MLaaS’s confidence in its results– that is available from MLaaS providers, but is typically ignored by software applications.Machine learning is now a major part of software systems that affect our daily lives, including transportation, medical systems, and even news distribution. The rise of MLaaS makes it even easier for non-experts to incorporate machine learning into these software systems, but it also increases the opportunities for a new class of software bugs and software failures. This project will identify and categorize the novel class of bugs that can arise from the use of MLaaS in larger software systems and create tools and methodologies to identify and fix those bugs. All benchmarks, data, and software tools developed through this project will be released as open source so that the larger community can freely benefit from this work. By improving the correctness and performance of software systems that use machine learning services, this project will not only make it easier to develop such software, but also improve the quality of people’s daily lives as the software they use will be more reliable.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.