Empirical and computational study of iconic gesture-speech integration and its development in preschool children (EcoGest)

手势和言语构成了一个完整的交际系统。为了交流和学习的目的，我们对这种融合进行了研究。最近试图解释和模拟这种整合的研究集中在手势如何影响言语，以及手势作为一种特殊的行为形式如何迫使说话者将感觉运动信息纳入表征。在这些认知方法中，忽略了这样一个事实，即手势被用于各种交际体裁，从而与言语出于不同的话语目的而相互作用。然而，各种类型的交际体裁对说话人和听话人有着不同的认知和交际要求：例如，叙述需要说话人不同的结构知识和语言技能，而不是解释。虽然对于有能力的说话者来说，不同交际体裁中的手势可能会流畅地出现，但对于学习如何应对各种话语语境的儿童来说，手势在思维和交流中的作用可能更容易观察到：一种可能性是，对于特定的体裁，手势是描述促进相互理解的策略所必需的资源，儿童随着互动经验的增加以及认知和语言能力的增强而发现它们。另一种可能性--不是排他性的--是手势在掌握不断变化的交际要求的过程中扮演着重要的角色，当儿童学习产生不同的交际类型时，这一点就变得明显起来。因此，除了用于说话行为的感觉运动信息外，手势可能是与不同交际目的有关的说话不可或缺的资源。这种可能性对手势-言语整合模式提出了新的问题：手势的使用依赖于交际体裁的经验吗？如果是这样的话，语用方面是如何融入言语产生的？在我们提出的项目中，我们应用定性和定量分析以及计算评估方法，纵向调查了4岁和5岁学龄前儿童手势和言语的整合是如何发展的，以及语境(即交际类型和物理环境)如何与儿童在互动中使用的不同手势技巧有关。我们项目的结果将对手势和言语整合的全面发展理论做出重要贡献，该理论涉及说话的认知和语用方面。

Gesture and speech form an integrated communicative system. This integration has been investigated for the purpose of communication as well as learning. Recent research attempting to explain and model this integration focus on how gesture influences speech, and how gesture -- as a special form of action -- forces the speaker to incorporate sensorimotor information into the representation. In these cognitive approaches, the fact that gesture is employed in various communicative genres and thus interacts with speech for different discursive purposes is disregarded. Yet types of communicative genres have been shown to bear different cognitive and communicative requirements on the speaker and hearer: For example, a narration requires different structural knowledge and verbal skills from a speaker than an explanation. While for competent speakers, the appearance of gestures in different communicative genres might occur fluent, for children learning to cope with the variety of discursive contexts, the role of gesture for thinking and communicating might be better observable: One possibility is that for particular genres, gestures are required resources for depicting strategies facilitating mutual understanding, and children discover them with increasing experience in interaction and with growing cognitive and linguistic competencies. Another -- not exclusive -- possibility is that gesture plays an important role on the way to master increasingly changing communicative requirements, which becomes visible when children learn to produce different communicative genres. Thus, in addition to sensorimotor information that is recruited for the act of speaking, gesture might be an integral resource of speaking in relation to different communicative purposes. This possibility raises new questions for models of gesture-speech integration: Is gesture used in dependence of the experience in communicative genre and if so how is the pragmatic aspect incorporated into the production of speech? In our proposed project, we apply qualitative and quantitative analyses as well as computational evaluation methods in investigating longitudinally how the integration of gesture and speech develops in 4 and 5 years old preschool children and how context (meaning the communicative genre and physical surroundings) relates to the different gestural techniques that children use in interaction. The results of our project will provide a crucial contribution towards a comprehensive developmental theory of gesture and speech integration involving cognitive and pragmatic aspects of speaking.