CAREER: Detecting, Understanding, and Fixing Vulnerabilities in Natural Language Processing Models

随着机器学习的最新进展，模型在自然语言处理(NLP)中的许多具有挑战性的任务上都获得了高精度，如问题回答、机器翻译和对话代理，有时在这些基准上接近或超过人类的表现。然而，这些NLP模型经常在许多不同的方面受到脆性的影响：它们抓住错误的人工产物，不支持语言的自然变化，对对手攻击不健壮，并且只在少数领域工作。现有的开发NLP模型的管道缺乏对有用见解的支持，识别错误需要机器学习和该领域的专家付出相当大的努力。这个职业项目开发了几种技术来支持对NLP更强大的培训和评估渠道的需求，为识别、理解和解决NLP模型的漏洞提供了易于使用、可扩展和准确的机制。开发的方法将支持不同的应用领域，如会话代理、情感分类器和辱骂/仇恨语音检测。此外，该团队与学术界和工业界的NLP模型开发人员合作，为K-12教育开发数据科学课程，特别是针对来自代表性不足社区的学生。基于脆弱性是某些输入转换的意外行为的概念，该团队将在以下三个方面做出贡献。第一个推力是通过测试用户定义的行为和搜索许多可能的漏洞来识别漏洞。在第二个推力中，调查人员开发了一些方法，通过追踪单个训练数据点和数据人工制品的错误原因来了解模型的漏洞。最后一项重点将开发解决模型中漏洞的方法，方法是在培训期间直接将漏洞定义注入模型中，并使用基于解释的注释来监督模型。这些努力建立在行为测试、基于解释的交互和架构不可知论的目标之上，以支持大多数当前和未来的NLP模型和应用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

With recent advances in machine learning, models have achieved high accuracy on many challenging tasks in natural language processing (NLP) such as question answering, machine translation, and dialog agents, sometimes coming close to or beating human performance on these benchmarks. However, these NLP models often suffer from brittleness in many different ways: they latch onto erroneous artifacts, do not support natural variations in language, are not robust to adversarial attacks, and only work on a few domains. Existing pipelines for developing NLP models lack support for useful insights, and identifying bugs requires considerable effort from experts both in machine learning and the domain. This CAREER project develops several techniques to support this need for more robust training and evaluation pipelines for NLP, providing easy-to-use, scalable, and accurate mechanisms for identifying, understanding, and addressing NLP models' vulnerabilities. The developed methods will support diverse application areas such as conversational agents, sentiment classifiers, and abuse/hate speech detection. Further, the team engages with the developers of NLP models in academia and industry to develop a data science curriculum for K-12 education, particularly for students from underrepresented communities.Based on the notion of vulnerability as unexpected behavior on certain input transformations, the team will contribute across the following three thrusts. The first thrust identifies vulnerabilities by testing user-defined behaviors and searching over many possible vulnerabilities. In the second thrust, the investigators develop methods to understand the model's vulnerabilities by tracing the causes of errors to individual training data points and data artifacts. The last thrust will develop approaches to address vulnerabilities in models by directly injecting the vulnerability definitions into the model during training and using explanation-based annotations to supervise the models. These thrusts build upon the goals of behavioral testing, explanation-based interactions, and architecture agnosticism to support most current and future NLP models and applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.