qaAskeR+\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^+$$\end{document}: a novel testing method for question answer
qaAskeR+\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^+$$\end{document}: a novel testing method for question answer
复制标题
DOI：
10.1007/s10515-023-00380-2
复制
发表时间：
2023-03
影响因子：
3.4
通讯作者：
Xiaoyuan Xie;Shuo Jin;Songqiang Chen
中科院分区：
文献类型：
--
作者：
Xiaoyuan Xie;Shuo Jin;Songqiang Chen
问答是自然语言处理领域中一个极具吸引力和挑战性的领域。随着问答技术的发展，大量的问答软件已经应用到人们的日常生活中，为人们提供了方便的信息检索途径。为了研究QA软件的性能，已经构建了许多基准数据集来提供各种测试用例。然而，当前的QA软件主要是在基于引用的范型中进行测试的，在这种范型中，测试用例的预期输出(标签)必须在测试前进行大量的人工标注。因此，无论是在使用过程中的即时测试，还是在海量未标记的真实数据上的可扩展测试都是不可行的，这使得目前的QA软件测试缺乏灵活性和充分性。在这项工作中，我们提出了一种新的测试方法qaAskeR，它具有五种新的变质关系，qaAskeR不是指测试用例的标注标签。相反，基于正确答案应该意味着总是与任何其他正确答案一致的可靠知识的想法，qaAskeRtest通过检查QA软件在与相同或某些进一步丰富的知识相关的多个递归提问上的行为来测试QA软件。实验结果表明，在不使用任何预先标注的标签的情况下，qaAskeR可以发现相当多的违规行为，这些违规行为表明了各种主流QA软件上的实际回答问题。
Question Answering (QA) is an attractive and challenging area in NLP community. With the development of QA technique, plenty of QA software has been applied in daily human life to provide convenient access of information retrieval. To investigate the performance of QA software, many benchmark datasets have been constructed to provide various test cases. However, current QA software is mainly tested in a reference-based paradigm, in which the expected outputs (labels) of test cases are mandatory to be annotated with much human effort before testing. As a result, neither the just-in-time test during usage nor the extensible test on massive unlabeled real-life data is feasible, which keeps the current testing of QA software from being flexible and sufficient. In this work, we propose a novel testing method,qaAskeR, with five new Metamorphic Relations for QA software.qaAskeRdoes not refer to the annotated labels of test cases. Instead, based on the idea that a correct answer should imply a piece of reliable knowledge that always conforms with any other correct answer,qaAskeRtests QA software by inspecting its behaviors on multiple recursively asked questions that are relevant to the same or some further enriched knowledge. Experimental results show thatqaAskeRcan reveal quite a few violations that indicate actual answering issues on various mainstream QA software without using any pre-annotated labels.