EAGER: Requirements Domain Specifications for Machine-Learned Software Components

EAGER：机器学习软件组件的需求领域规范

• 批准号: 2124606
• 负责人: Mona Rahimi
• 金额: $ 14.64万
• 依托单位: Northern Illinois University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-04-01 至 2023-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2124606&HistoricalAwards=false
• 关键词: EAGER; Requirements; Domain; Specifications; Machine

Software components are traditionally built according to a set of pre-defined specifications that define what a software component is expected to do in a given context. These specifications are traditionally gathered from stakeholders and customers during the domain-analysis and requirements-engineering phases. In contrast to deterministic software components, software components built using Machine-Learning (ML) algorithms learn their specifications from a set of collected examples rather than a set of agreed upon specifications. When the functional correctness of ML-enabled software depends only on the training data, there can be a significant gap between specification of a real-world concept and what a collected dataset represents as the concept. The goal of this research is to define the meaning of requirements satisfaction for software with machine-learning components, and to investigate methods for engineering those requirements. In this project, the investigators will formally specify partial requirements for Machine-Learned Components (MLCs) instead of allowing them to learn these specifications solely from a set of collected samples in an ad-hoc manner. The goal is thus to make machine-learning components better meet requirements by augmenting the inductive nature of ML with domain analysis, in order to characterize the extent to which the dataset contains or lacks important features that are necessary to meet requirements. This project provides a framework for formally specifying partial requirements as well as validating the presence of such specifications in the collected samples, which in essence characterizes the extent to which the dataset contains or lacks features important to learning the task. The proposal considers this problem in the context of automated driving systems where the correct definition of real-world concepts is critically important for safety reasons. For example, correct image recognition is needed to classify objects to avoid; the objective is to show that combining training by datasets with partial domain models from elicited requirements can outperform brute-force ML.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

传统上，软件组件是根据一组预定义的规范构建的，这些规范定义了软件组件在给定上下文中的预期功能。 这些规范传统上是在领域分析和需求工程阶段从涉众和客户那里收集的。与确定性软件组件相比，使用机器学习（ML）算法构建的软件组件从一组收集的示例而不是一组商定的规范中学习它们的规范。当支持ML的软件的功能正确性仅取决于训练数据时，在现实世界概念的规范与收集的数据集表示的概念之间可能存在显著差距。本研究的目标是定义的意义，需求满足的软件与机器学习组件，并调查工程的方法，这些要求。 在该项目中，研究人员将正式指定机器学习组件（MLC）的部分要求，而不是允许他们仅从一组收集的样本中学习这些规范。 因此，我们的目标是通过域分析增强ML的归纳性质，使机器学习组件更好地满足要求，以表征数据集包含或缺乏满足要求所需的重要特征的程度。该项目提供了一个框架，用于正式指定部分需求以及验证收集的样本中是否存在此类规范，这在本质上表征了数据集包含或缺乏对学习任务重要的功能的程度。该提案在自动驾驶系统的背景下考虑了这个问题，其中出于安全原因，正确定义真实世界概念至关重要。例如，需要正确的图像识别来对物体进行分类以避免;其目标是表明将数据集训练与引发需求的部分域模型相结合可以胜过蛮力ML。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

CADE: The Missing Benchmark in Evaluating Dataset Requirements of AI-enabled Software

• DOI: 10.1109/re54965.2022.00013
• 发表时间: 2022-08
• 期刊: 2022 IEEE 30th International Requirements Engineering Conference (RE)
• 作者: H. Barzamini;Mona Rahimi

B-AIS: An Automated Process for Black-box Evaluation of Visual Perception in AI-enabled Software against Domain Semantics

• DOI: 10.1145/3551349.3561162
• 发表时间: 2022-10
• 期刊: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
• 作者: H. Barzamini;Mona Rahimi

Improving Generalizability of ML-enabled Software through Domain Specification

• DOI: 10.1145/3522664.3528589
• 发表时间: 2022-05
• 期刊: 2022 IEEE/ACM 1st International Conference on AI Engineering – Software Engineering for AI (CAIN)
• 作者: H. Barzamini;Mona Rahimi;Murteza Shahzad;Hamed Alhoori

A multi-level semantic web for hard-to-specify domain concept, Pedestrian, in ML-based software

• DOI: 10.1007/s00766-021-00366-0
• 发表时间: 2022-01
• 期刊: Requirements Engineering
• 影响因子: 2.8
• 作者: H. Barzamini;Murtuza Shahzad;Hamed Alhoori;Mona Rahimi