REU Site: Animal Language Processing and Understanding

动物在谈论什么是一个令人着迷的研究主题。之前试图了解动物之间的交流依赖于行为观察，而且只研究了少数几个物种，成本很高。这些尝试都没有采用一种计算和系统的方法，将动物的声音与书面符号和意义联系起来，这是我们通常对人类语言所做的。这个项目使用当前先进的机器学习方法，调查动物是否有像人类一样在句法和语义上构成的语言。在一项初步研究中，该团队开发了一条“动物语言处理”管道，对大量关于家犬的YouTube视频进行清理和预处理，并进一步将狗狗发声的音频片段转录成一系列音标。在这个项目中，本科生研究人员将收集高质量的、部分注释的多媒体动物交流数据，将其输入前述管道，分析得到的成绩单，并对新物种的语言获得新的见解。这项由本科生研究人员开展的工作还将发展成为一个名为AniVoice的开源网络动物语言研究平台。这个项目的结果将是朝着回答上面提到的大科学问题迈出的一步，并最终有助于更好地了解我们周围的世界。这个项目还将支持本科生参与者发展跨学科研究的专门知识。这个项目包括技术里程碑。首先，研究人员获得了目标动物物种的高质量、干净的发声数据，部分注释了动物参与的活动或场景。数据是通过现场录制或从互联网下载相关视频片段来收集的。其次，研究人员将动物交流的音频文件分割成最小的可听声音单元，很像人类语音的音节，然后将常规的动物音节序列分组为动物单词。这带来了巨大的挑战，因为人类不知道如何解析动物的交流。第三，研究人员开发了模型，通过观察动物在发出某个单词时的活动或位置，自动理解单词的语义。这就需要实现视频理解算法，特别是场景和活动分类，以及在存在多个动物的场景中的活动说话人检测。最后，本科生研究人员实现了一个基于网络的众包界面，动物爱好者或研究人员可以在其中上传他们的录音，使用提供的工具注释他们的数据，并评估来自动物语言处理管道的转录和语义分析结果。AniVoice平台将被用作未来推进动物语言研究的工具。在这个项目的每一年，一批本科生研究人员都将通过不断改进处理管道、在更多动物物种上添加更多实验证据以及改进AniVoice网络服务来做出贡献。由学生研究人员进行的分析有望为目标物种的语言能力提供更多的科学证据，这可能会带来生物学家和行为生态学家的新见解。AniVoice为动物通信方面的数据驱动和跨学科研究合作打开了机会。从这个项目中获得的知识也可能对研究古代或灭绝的人类语言有用。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

What animals are talking about is a fascinating research topic. Previous attempts to understand animal communications relied on behavioral observations, and have studied only a handful of species given high costs. None of these attempts have adopted a computational and systematic approach to associate animal vocal sounds to written symbols and meanings as we typically do with human languages. This project investigates whether animals have languages that are syntactically and semantically compositional like humans, using current advanced machine learning methodology. In a preliminary study, the team has developed an "animal language processing" pipeline that cleans and preprocesses a large amount of YouTube videos about domestic dogs, and further transcribes the audio segments of dog vocalizations into a sequence of phonetic symbols. In this project, undergraduate researchers will collect high-quality, partially annotated multimedia animal communication data, feed it into the aforementioned pipeline, analyze the resulting transcripts and gain new insights on the languages of new species. The work conducted by the undergraduate researchers will also develop into an open-source web-based animal language study platform called AniVoice. The outcome of this project will be a step toward answering the big science question noted above, and eventually help better understand the world around us. This project will also support the undergraduate participants in developing a expertise in interdisciplinary research.This project includes technical milestones. First, the researchers acquire high-quality, cleaned vocalization data of a target animal species, partially annotated with the activity or scenario the animal is involved in. Data is collected either by live recording, or downloading relevant video clips from the internet. Second, the researchers segment the audio files of animal communications into minimum units of audible sounds, much like the syllables of human speech, and then group regular animal syllable sequences into animal "words". This poses significant challenges because human beings do not know how to parse animal communications. Third, the researchers develop models to automatically understand the semantics of the words by looking at the activities or location of the animal when a certain word is uttered. This requires the implementation of video understanding algorithms, and in particular, scene and activity classification, as well as "active speaker" detection in a scene where multiple animals exist. Finally, the undergraduate researchers implement a web-based crowdsource interface where animal lovers or researchers can upload their recordings, annotate their data using the tools provided, and evaluate the transcription and semantic analysis results from the animal language processing pipeline. The AniVoice platform will be used as a tool to advance animal language studies in the future. Each year of this project, a cohort of undergraduate researchers will contribute by continuously improving the processing pipeline, adding more experimental evidence on more animal species, and improving the AniVoice web service. The analysis performed by the student researchers is expected to provide additional scientific evidence of the language abilities of target species, which may lead to new insights by biologists and behavioral ecologists. AniVoice opens opportunities for data-driven and interdisciplinary research collaboration on animal communications. The knowledge gained in this project may also be useful in the study of ancient or extinct human languages.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.