Approaching 100 Percent Recall for Requirements and Software Engineering Tools

点击翻译按钮获取中文摘要

A hairy requirements or software engineering task involving natural language (NL) documents is one that is not inherently difficult for NL understanding humans on a small scale but becomes unmanageable in the large scale. Examples include identification of abstractions, ambiguities, synonyms, and trace links. A hairy task demands tool assistance. Because humans need far more help in carrying out a hairy task completely than they do in making the local yes-or-no decisions, a tool for a hairy task should have as close to 100% recall (that the tool finds all desired information) as possible, even at the expense of high imprecision (that not all the information that the tool finds is desired). A tool that falls much short of 100% recall may even be useless, because to find the missing desired information, a human has to do the entire task manually anyway. Any such tool based on NL processing (NLP) techniques inherently fails to achieve 100% recall, because even the best parsers are no more than 91% correct. Therefore, for a tool to achieve 100% recall for a hairy task, it needs to be based on something other than traditional NLP. The reality is that a tool's achieving exactly 100% recall, which may be impossible anyway, may not be necessary. It suffices for a human working with the tool on a task to achieve better recall than a human working on the task entirely manually. The proposed research is to discover and test a variety of non-traditional approaches to building tools for hairy tasks to see which, if any, allows a human working with the tool to achieve better recall than a human working entirely manually. If the research succeeds, we will be able to build tools for hairy tasks that demonstrably out-perform humans on these tasks. Therefore, when a software or requirements engineer is faced with one of these hairy tasks, he or she will trust the completeness of the output of the tool and will not feel compelled to do the same task manually.