FAI: A New Paradigm for the Evaluation and Training of Inclusive Automatic Speech Recognition

自动语音识别可以在一些小方面提高您的工作效率：使用自动语音识别完成这些任务通常比使用图形用户界面搜索歌曲、产品或地址更快。然而，对于许多人群来说，语音识别的效果不太好，可能是因为地区口音，或者是因为第二语言口音，或者是因为残疾。这个AI公平性项目定义了一种思考语音技术的新方式。按照这种新的思维方式，除非自动语音识别器适用于所有用户，包括具有地区口音、第二语言口音和严重残疾的用户，否则它不能很好地工作。有三个子项目。第一个子项目将创建黑盒测试标准，语音技术研究人员可以用它来测试他们的语音识别器，以测试他们的语音识别器对不同人群的有用程度。例如，如果研究人员发现他们的产品对某些人有效，但对其他人无效，那么研究人员将有机会收集更多的培训数据，并进行更多的开发，以确保服务不足的社区得到更好的服务。第二个子项目将创建玻璃盒测试标准，研究人员可以用它来调试包容性问题。例如，如果语音识别器在识别特定方言时遇到困难，那么玻璃盒方法将识别该方言中使识别器困惑的特定语音，以便研究人员可以更有效地解决问题。第三个子项目将创建训练语音识别器的新方法，以保证它对可用数据中表示的所有不同组都能很好地工作。数据将来自播客和互联网。当且仅当发言者声明自己是某一特定小组的成员时，他们将被认定为该小组的成员。所有开发的软件都将是开源的。自动语音识别有可能使信息流民主化：人工智能对话代理可以为那些不知道该往哪里看的人提供信息。在过去的五十年里，语音开发社区对最低错误率的不懈关注导致了一个生产力工具，对于那些语音模式与其训练数据相匹配的人来说，它工作得非常好：通常是受过大学教育的第一语言使用者，使用一种标准化的方言，很少或没有语言障碍。然而，对于许多人群来说，语音识别效果不太好，可能是因为他们的语音模式与标准方言有很大不同（例如，由于地区口音），可能是因为群体内的异质性（例如，非洲裔美国人的地区方言），或者是因为群体中每个人的语音模式都表现出可变性（例如，严重残疾的人或第二语言学习者）。本提案的目的是为包容性自动语音识别器的评估和训练创造一个新的范例。提出的新的评估和训练范式由三个部分组成：(1)“黑箱评估”是一种评估，可以通过观察语音识别器的输出来衡量其包容性程度，而无需访问源代码或训练参数。使用适当平衡的测试数据，统计测试可以确定系统是否为所有用户组提供相同的错误率，如果不同组获得不同的错误率，则可以将差异的大小读取为问题大小的度量。(2)“玻璃盒评估”是一种评估，它识别出在不同组之间始终存在的错误模式，并在声信号和网络的训练参数中寻找这些错误的原因。(3)包容性优化是一系列端到端的神经网络训练标准，以及训练数据集设计和增强标准，它们明确地平衡了低平均错误率与低组间和说话人间方差的需求。为了开发这些新的评估和培训范例，研究人员建议开发和分发开源数据和工具。数据将来自大型公共数据源，包括10万个播客语料库；研究人员将在语料库中搜索对话行为，其中说话者将自己与特定群体联系起来，然后将发现的群体身份和人工转录作为开源元数据分发。工具将使用包括K2在内的开源工具包来实现，这些工具将作为开源系统食谱分发。语音技术开发人员是一群竞争激烈的人：如果有一个数字来描述语音识别器的包容性，如果有理由相信这个数字在科学上是有根据的和可取的，那么世界各地的研究人员将竞争使他们的系统更具包容性。拟议的研究将开发这样的指标和相关数据，并将其部署为开源。这项研究将作为人工智能的社会影响的模型，在正在进行的扩展计划的调查人员。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Automatic speech recognition can improve your productivity in small ways: rather than searching for a song, a product, or an address using a graphical user interface, it is often faster to accomplish these tasks using automatic speech recognition. For many groups of people, however, speech recognition works less well, possibly because of regional accents, or because of second-language accent, or because of a disability. This Fairness in AI project defines a new way of thinking about speech technology. In this new way of thinking, an automatic speech recognizer is not considered to work well unless it works well for all users, including users with regional accents, second-language accents, and severe disabilities. There are three sub-projects. The first sub-project will create black-box testing standards that speech technology researchers can use to test their speech recognizers, in order to test how useful their speech recognizer will be for different groups of people. For example, if a researcher discovers that their product works well for some people, but not others, then the researcher will have the opportunity to gather more training data, and to perform more development, in order to make sure that the under-served community is better-served. The second sub-project will create glass-box testing standards that researchers can use to debug inclusivity problems. For example, if a speech recognizer has trouble with a particular dialect, then glass-box methods will identify particular speech sounds in that dialect that are confusing the recognizer, so that researchers can more effectively solve the problem. The third sub-project will create new methods for training a speech recognizer in order to guarantee that it works equally well for all of the different groups represented in available data. Data will come from podcasts and the Internet. Speakers will be identified as members of a particular group if and only if they declare themselves to be members of that group. All of the developed software will be distributed open-source.Automatic speech recognition has the potential to democratize the flow of information: artificially intelligent dialog agents can provide information to people who would otherwise not know where to look. The speech developer community's relentless focus on minimum error rate over the past fifty years has resulted in a productivity tool that works extremely well for those of whose speech patterns match its training data: typically, college-educated first-language speakers of a standardized dialect, with little or no speech disability. For many groups of people, however, speech recognition works less well, possibly because their speech patterns differ significantly from the standard dialect (e.g., because of regional accent), because of intra-group heterogeneity (e.g., regional African American dialects), or because the speech pattern of each individual in the group exhibits variability (e.g., people with severe disabilities, or second-language learners). The aim of this proposal is to create a new paradigm for the evaluation and training of inclusive automatic speech recognizers. The proposed new evaluation and training paradigm consists of three components: (1) A "black-box evaluation" is an evaluation that can measure the degree of inclusivity of a speech recognizer by observing its outputs, without access to source code or trained parameters. With appropriately balanced test data, a statistical test can determine whether or not a system provides all groups of users with the same error rates, and if different groups get different error rates, then the size of the difference can be read as a measurement of the size of the problem. (2) A "glass-box evaluation" is an evaluation that identifies error patterns that consistently differentiate between groups, and searches for the causes of those errors in the acoustic signal and in the trained parameters of the network. (3) Inclusive optimization is a family of end-to-end neural network training criteria, and training dataset design and augmentation criteria, that explicitly balance the need for low average error rate against the need for low inter-group and inter-speaker variance. In order to develop these new evaluation and training paradigms, the researchers propose to develop and distribute open-source data and tools. Data will be drawn from large public data sources including the 100,000-podcast corpus; researchers will search the corpus for dialog acts in which speakers identify themselves with a particular group, then distribute discovered group identities and manual transcriptions as open-source metadata. Tools will be implemented using open source toolkits including K2, and those tools will be distributed as open-source system recipes. Speech technology developers are a competitive bunch: if there is a single number that describes the inclusivity of a speech recognizer, and if there is reason to believe that number to be scientifically well-founded and desirable, then researchers all over the world will compete to make their systems more inclusive. Proposed research will develop such metrics, and associated data, and will deploy them open-source. This research will be held up as a model of the social impact of artificial intelligence in the ongoing outreach programs of the investigators.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.