A Study of Multimodal Behavior Corpus for Acquiring Infants' Fundamental Commonsense

本研究建立了一个“多模态婴儿行为语料库”，为人类基本常识的阐释做出了宝贵的贡献。在2005年，我们开发了语料库的基本部分，并开始实际使用它通过观察多模态数据记录在环境中的父母和婴儿的混合教育。2006年，我们建立了一种方法，通过婴儿行为记录、行为分析、婴儿教育材料开发和婴儿课堂实践评估的循环来扩展语料库。在2007年，我们讨论了为构建常识性知识而对思维和行为之间的关系进行建模的结构。我们还研究了如何扩展语料库以及如何将开发的录音环境应用到其他领域。以下是实际研究计划中每个主题的详细结果。(1)录音环境设计：我们建立了一个幼儿学校，每周举行一次。 ...更多信息 y.所有课程都以录像数据和硬盘形式记录下来。语音数据也与视频同时存储。整理录音材料的环境，构建语料库。我们还研究了如何将记录环境应用于支持通信的工具，如提供可访问的信息，并确认开发的环境可以用于各种领域。(2)行为语料库的构建：我们引入了一种可穿戴的语音记录设备，并从婴儿的行为和话语数据中产生视频内容。我们最初开发了一个注释支持工具，并实现了一个系统，通过定期举行的会议上的讨论，建立了一个方法来构建语料库的注释进行调查。(3)幼儿教育内容的创造：幼儿学校教师与行为分析研究者合作，不断增加教育材料。我们实现了一个由以下步骤组成的教育内容的创建，实践和评估的循环系统：在使用材料的场景中分析幼儿的行为，指定不足的材料，创建教育材料，并将其引入幼儿课堂。(4)常识知识提取：我们根据Marvin Minsky提出的多层次思维模式进行了行为分析，获得了关于婴儿思维与其行为之间复杂关系的有价值的发现。在常识知识提取方面，我们针对手势、注视方向、话语等不同形态建立了幼儿指向行为的发展过程模型。通过以上多方面的研究，我们不仅实现了最初的目标，而且为婴儿行为模拟系统在真实的世界中的应用探索了一条道路。少

In this project, we have constructed a "Multimodal Infant Behavior Corpus," which constitutes a valuable contribution to the elucidation of the human fundamental commonsense knowledge. In 2005, we developed the basic parts of the corpus and started the practical use of it by observing multimodal data recorded in the environments for coeducation of parents and infants. In 2006, we built a methodology for expanding the corpus spirally through the cycle of infant behavior recording, behavior analysis, infant education material development, and practical evaluation in infant classes. In 2007, we discussed the structures for modeling the relationships between thinking and behavior to construct commonsense knowledge. We also studied how to extend the corpus and how to apply the developed recording environments to other domains. The following shows the results in detail for each subject in the practical research plan.(1) Recording environment design: We established an infant school held weekl … More y. All classes had been recorded as video data and on hard discs. Speech data was also stored simultaneously with the video. We arranged the environment for recording materials to construct the corpus. We also studied how to apply the recording environment to the tools for supporting communication such as providing accessible information, and confirmed that the developed environment was possible to be utilized in various domains.(2) Behavior corpus construction: We introduced a wearable speech recording device and produced video contents from infant behavior and utterance data. We originally developed an annotation support tool and realized a system for investigating the annotations by discussions at the conferences held regularly, which established a methodology for constructing the corpus.(3) Contents creation for infant education: The teachers of the infant school and the researchers of behavior analysis had collaboratively increased educating materials constantly. We realized a cyclic system for creating, practicing and evaluating the educating contents consisted of the following steps; analysis of infant behavior in the scenes of using the materials, specification of insufficient materials, creation of educating materials, and introduction of them to the infant classes.(4) Commonsense knowledge extraction: We conducted behavior analysis based on the multilayered thinking model suggested by Marvin Minsky, and obtained valuable findings about the complicated relations between infant thinking and its behavior. Toward commonsense knowledge extraction, we made the models of infant developing process in pointing behavior for each modality such as gesture, direction of eye gaze, and utterance. We developed an infant behavior simulation system by integrating the models and investigated the validity of them.Through the above manifold studies, we have achieved not only the original goals, but also explored a route for the application to the real world. Less