CompCog: Deep causal inference grounds the perception of cognitive objects in speech

人工智能系统在社会中变得越来越重要，近年来它们的性能有了很大的提高。然而，我们仍然不了解这些系统实际上是如何工作的，以及它们如何模仿人类的表现。在这项工作中，开发了新的方法来探测这些系统的内部工作，通过比较它们的内部计算与人类执行的相应计算。我们探讨的具体技能是语音识别-一个高度复杂的过程，因为语音是一个丰富的变量，信息密集，传播迅速的沟通媒介。人类大脑在语音识别过程中处理这种复杂性的方法之一是不仅参与负责倾听的大脑区域，而且参与对语音产生至关重要的区域。这表明人类的认知意识到了世界上引起言语的系统--嘴唇、舌头和其他发声器官的运动。当前的人工智能系统是否也发展出了对语音的如此深刻的因果理解？在这项工作中，我们通过深入研究这些系统中人类和机器知识的数学模型来回答这个问题。我们的工作有两个主要目标。首先是技术：了解人工智能系统内部实际上是如何工作的，这最终是将其能力用于社会利益的必要步骤。二是科学性：在广泛应用于社会之前，人工智能系统是由认知科学家开发的，以了解人类的认知，通过探索最先进的机器学习作为认知模型的内部工作原理，我们可能能够更好地了解人类如何感知语音。因此，在这项工作中，科学和技术相互促进，因为他们已经成功地在过去做了。这个研究计划专门探讨人类和计算机的语音生产和感知之间的关系。为此，使用真实的时间磁共振成像（MRI）对三种语言（英语、俄语、韩语）的说话者进行成像，该成像生动详细地显示了语音发音器官如何移动。 说话者的语音音频信号被同时记录。使用语音产生的数学模型，现代语音识别系统和人类神经节律如何分析语音的数学模型来分析数据。实验操作揭示了每个系统中的表示如何与其他系统中的表示相对应。这一策略在揭示人类认知科学的同时，也揭示了机器模拟人类能力的黑箱技术。今后，除了推进科学技术的发展外，我们还期待将这些知识应用于创建新型小型语音识别系统，以帮助记录濒危语言。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial Intelligence systems are becoming more and more important in society, and their performance has improved enormously in recent years. Yet, we still do not understand how these systems actually work, and how they emulate human performance, to the extent that they do. In this work, novel methods are developed for probing the inner working of these systems by comparing their internal computations with corresponding computations humans perform. The specific skill we probe is speech recognition—a highly complex process, as speech is a richly variable, information dense, and quickly transmitted medium of communication. One of the ways that the human brain deals with this complexity during speech recognition is by engaging not only brain areas responsible for listening but also areas crucial to the production of speech. This suggests that human cognition is aware of the systems in the world that cause speech—the movements of the lips, tongue, and other vocal organs. Do current artificial intelligence systems also develop such a deep causal understanding of speech? In this work we answer this question by delving into the mathematical models of both human and machine knowledge in these systems. Our work has two major goals. The first is technological: understanding how artificial intelligence systems actually work on the inside, which is ultimately a necessary step in directing their abilities to societal benefit. The second is scientific: before widespread use in society, artificial intelligence systems were developed by cognitive scientists to understand human cognition, and by probing the inner workings of state-of-the-art machine learning as a cognitive model we may be able to better understand how humans perceive speech. In this work, therefore, science and technology further each other, as they have done successfully in the past.This research program specifically probes the relationship between the production and perception of speech in humans and computers. To do so, speakers of three languages (English, Russian, Korean) are imaged using a real time Magnetic Resonance Imaging (MRI), which shows in vivid detail how the speech articulators move. Speaker’s speech audio signals are recorded simultaneously. The data are analyzed using mathematical models of speech production, modern speech recognition systems, and mathematical models of how human neural rhythms analyze speech. Experimental manipulations unveil how the representations in each of the systems corresponds to those in the others. This strategy inform us about the science of human cognition hand-in-hand with illuminating the black-box technology of machine emulation of the human capability. In the future, in addition to advancing science and technology, we anticipate the application of this knowledge to the creation of novel small-sized speech recognition systems that can assist in the documentation of endangered languages.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.