A Study of Multimodal Behavior Corpus for Acquiring Infants' Fundamental Commonsense

点击翻译按钮获取中文摘要

In this project, we have constructed a "Multimodal Infant Behavior Corpus," which constitutes a valuable contribution to the elucidation of the human fundamental commonsense knowledge. In 2005, we developed the basic parts of the corpus and started the practical use of it by observing multimodal data recorded in the environments for coeducation of parents and infants. In 2006, we built a methodology for expanding the corpus spirally through the cycle of infant behavior recording, behavior analysis, infant education material development, and practical evaluation in infant classes. In 2007, we discussed the structures for modeling the relationships between thinking and behavior to construct commonsense knowledge. We also studied how to extend the corpus and how to apply the developed recording environments to other domains. The following shows the results in detail for each subject in the practical research plan.(1) Recording environment design: We established an infant school held weekl … More y. All classes had been recorded as video data and on hard discs. Speech data was also stored simultaneously with the video. We arranged the environment for recording materials to construct the corpus. We also studied how to apply the recording environment to the tools for supporting communication such as providing accessible information, and confirmed that the developed environment was possible to be utilized in various domains.(2) Behavior corpus construction: We introduced a wearable speech recording device and produced video contents from infant behavior and utterance data. We originally developed an annotation support tool and realized a system for investigating the annotations by discussions at the conferences held regularly, which established a methodology for constructing the corpus.(3) Contents creation for infant education: The teachers of the infant school and the researchers of behavior analysis had collaboratively increased educating materials constantly. We realized a cyclic system for creating, practicing and evaluating the educating contents consisted of the following steps; analysis of infant behavior in the scenes of using the materials, specification of insufficient materials, creation of educating materials, and introduction of them to the infant classes.(4) Commonsense knowledge extraction: We conducted behavior analysis based on the multilayered thinking model suggested by Marvin Minsky, and obtained valuable findings about the complicated relations between infant thinking and its behavior. Toward commonsense knowledge extraction, we made the models of infant developing process in pointing behavior for each modality such as gesture, direction of eye gaze, and utterance. We developed an infant behavior simulation system by integrating the models and investigated the validity of them.Through the above manifold studies, we have achieved not only the original goals, but also explored a route for the application to the real world. Less