CSR: Medium: Improving the Interface between Machine Learning and Software Systems

机器学习即服务(MLaaS)是一种新的软件范式，它使开发人员能够访问在海量数据集上训练的强大的机器学习模型，而不需要这些开发人员知道如何训练模型或访问所需的训练数据。这种方法使机器学习可以被更广泛的软件系统访问，但它也带来了新的挑战。具体地说，在提供非常通用的MLaaS接口(使其尽可能广泛地适用)的愿望和使用MLaaS的个别应用程序的特定需求之间存在着矛盾。例如，许多MLaaS提供商将通用对象检测模型作为服务提供，它可以识别图片中数以万计的不同对象，但大多数应用程序只需要该功能的一小部分；例如，与交通有关的应用程序只关心可能出现在道路上的对象。本项目将探讨这一矛盾--保留MLaaS的通用性，同时提高使用这些服务的个别应用程序的健壮性、准确性和性能。具体地说，该项目将首先创建一个真实世界应用程序的基准套件，以推动对由于一般MLaaS接口和特定应用程序需求之间的紧张关系而产生的软件错误进行经验研究。在这项研究的基础上，该项目将创建一套工具，自动调整软件，以修复由于在特定应用程序环境中使用通用MLaaS接口而产生的不一致和模棱两可。最后，该项目将创建重构软件的方法和工具，以使用额外的信息-包括MLaaS对其结果的信心-这些信息可以从MLaaS提供商那里获得，但通常被软件应用程序忽略。机器学习现在是影响我们日常生活的软件系统的主要部分，包括交通、医疗系统，甚至新闻发布。MLaaS的兴起使得非专家将机器学习融入这些软件系统变得更加容易，但它也增加了一类新的软件错误和软件故障的机会。该项目将识别和分类在大型软件系统中使用MLaaS可能产生的新型错误，并创建工具和方法来识别和修复这些错误。通过该项目开发的所有基准、数据和软件工具将作为开源发布，以便更大的社区可以自由地从这项工作中受益。通过改进使用机器学习服务的软件系统的正确性和性能，该项目不仅将使开发此类软件变得更容易，还将改善人们的日常生活质量，因为他们使用的软件将更加可靠。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Machine-Learning-as-a-Service (MLaaS) is a new software paradigm that gives developers access to powerful machine learning models trained on massive data sets without requiring those developers know how to train the models or have access to the required training data. This approach makes machine learning accessible to a much wider range of software systems, but it also creates new challenges. Specifically, there is a tension between the desire to provide a very general MLaaS interface (to make it as widely applicable as possible) and the specific needs of individual applications that use MLaaS. For example, many MLaaS providers offer general object detection models as a service, which can recognize tens of thousands of different objects in a picture, but most applications require only a small subset of that capability; e.g., applications concerned with traffic only care about objects that could appear on a roadway. This project will explore this tension – to preserve the generality of MLaaS while improving the robustness, accuracy, and performance of individual applications that use these services. Specifically, the project will first create a benchmark suite of real-world applications to drive an empirical study of the software bugs that arise due to the tension between general MLaaS interfaces and specific application needs. Based on that study, the project will create a set of tools that automatically adapt software to fix inconsistencies and ambiguities that arise due to the use of general MLaaS interfaces in application-specific contexts. Finally, the project will create methods and tools for refactoring software to use additional information –including the MLaaS’s confidence in its results– that is available from MLaaS providers, but is typically ignored by software applications.Machine learning is now a major part of software systems that affect our daily lives, including transportation, medical systems, and even news distribution. The rise of MLaaS makes it even easier for non-experts to incorporate machine learning into these software systems, but it also increases the opportunities for a new class of software bugs and software failures. This project will identify and categorize the novel class of bugs that can arise from the use of MLaaS in larger software systems and create tools and methodologies to identify and fix those bugs. All benchmarks, data, and software tools developed through this project will be released as open source so that the larger community can freely benefit from this work. By improving the correctness and performance of software systems that use machine learning services, this project will not only make it easier to develop such software, but also improve the quality of people’s daily lives as the software they use will be more reliable.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.