A Study of Multimodal Behavior Corpus for Acquiring Infants' Fundamental Commonsense

在这个项目中，我们构建了一个“多模态婴儿行为语料库”，它对阐明人类基本常识知识做出了有价值的贡献。2005年，我们开发了语料库的基本部分，并通过观察在父母和婴儿同校环境中记录的多模态数据开始了它的实际应用。2006年，我们建立了一种方法，通过婴儿行为记录、行为分析、婴儿教育材料开发和婴儿课堂实践评估的循环来螺旋式扩展语料库。2007年，我们讨论了思维和行为之间关系的建模结构，以构建常识性知识。我们还研究了如何扩展语料库以及如何将开发的记录环境应用于其他领域。以下是实际研究计划中各课题的详细结果。(1)录制环境设计：我们建立了每周一次的幼儿学校…More y.所有的课程都以视频数据和硬盘的形式记录下来。语音数据也与视频同时存储。我们安排了记录材料的环境来构建语料库。我们还研究了如何将记录环境应用于支持通信的工具，例如提供可访问的信息，并确认了开发的环境可以在各种领域中使用。(2)行为语料库构建：引入可穿戴语音记录设备，根据婴儿的行为和话语数据制作视频内容。我们最初开发了一个标注支持工具，并通过定期召开的会议讨论实现了一个标注调查系统，建立了语料库的构建方法。(3)幼儿教育内容创新：幼儿学校教师与行为分析研究者共同不断增加教材。我们实现了一个由以下几个步骤组成的教育内容创造、实践和评价的循环系统；分析幼儿在使用材料场景下的行为，规范材料不足，创作教材，并将教材引入幼儿课堂。(4)常识性知识提取：基于Marvin Minsky提出的多层思维模型进行行为分析，获得了关于婴幼儿思维与行为之间复杂关系的有价值的发现。针对常识性知识的提取，我们针对手势、目光注视方向、话语等不同的情态建立了婴儿指向行为的发展过程模型。我们将这些模型整合在一起，开发了一个婴儿行为模拟系统，并对其有效性进行了研究。通过以上多方面的研究，我们不仅达到了最初的目标，而且探索出了一条应用于现实世界的途径。少

In this project, we have constructed a "Multimodal Infant Behavior Corpus," which constitutes a valuable contribution to the elucidation of the human fundamental commonsense knowledge. In 2005, we developed the basic parts of the corpus and started the practical use of it by observing multimodal data recorded in the environments for coeducation of parents and infants. In 2006, we built a methodology for expanding the corpus spirally through the cycle of infant behavior recording, behavior analysis, infant education material development, and practical evaluation in infant classes. In 2007, we discussed the structures for modeling the relationships between thinking and behavior to construct commonsense knowledge. We also studied how to extend the corpus and how to apply the developed recording environments to other domains. The following shows the results in detail for each subject in the practical research plan.(1) Recording environment design: We established an infant school held weekl … More y. All classes had been recorded as video data and on hard discs. Speech data was also stored simultaneously with the video. We arranged the environment for recording materials to construct the corpus. We also studied how to apply the recording environment to the tools for supporting communication such as providing accessible information, and confirmed that the developed environment was possible to be utilized in various domains.(2) Behavior corpus construction: We introduced a wearable speech recording device and produced video contents from infant behavior and utterance data. We originally developed an annotation support tool and realized a system for investigating the annotations by discussions at the conferences held regularly, which established a methodology for constructing the corpus.(3) Contents creation for infant education: The teachers of the infant school and the researchers of behavior analysis had collaboratively increased educating materials constantly. We realized a cyclic system for creating, practicing and evaluating the educating contents consisted of the following steps; analysis of infant behavior in the scenes of using the materials, specification of insufficient materials, creation of educating materials, and introduction of them to the infant classes.(4) Commonsense knowledge extraction: We conducted behavior analysis based on the multilayered thinking model suggested by Marvin Minsky, and obtained valuable findings about the complicated relations between infant thinking and its behavior. Toward commonsense knowledge extraction, we made the models of infant developing process in pointing behavior for each modality such as gesture, direction of eye gaze, and utterance. We developed an infant behavior simulation system by integrating the models and investigated the validity of them.Through the above manifold studies, we have achieved not only the original goals, but also explored a route for the application to the real world. Less