Empirical and computational study of iconic gesture-speech integration and its development in preschool children (EcoGest)

手势和言语构成了一个完整的交际系统。为了交流和学习的目的，研究了这种整合。最近的研究试图解释和模拟这种整合，重点是手势如何影响语言，以及手势作为一种特殊的动作形式如何迫使说话者将感觉运动信息整合到表征中。在这些认知方法中，手势在各种交际类型中被使用，从而与言语相互作用以达到不同的话语目的，这一事实被忽视了。然而，不同类型的交际体裁对说话者和听话者的认知和交际要求是不同的：例如，叙述对说话者的结构知识和语言技能的要求不同于解释。虽然对于有能力的说话者来说，手势在不同交际类型中的出现可能会很流畅，但对于学习应对各种话语环境的儿童来说，手势在思考和交流中的作用可能会更明显：一种可能性是，对于特定类型，手势是描述促进相互理解的策略所需的资源，儿童随着互动经验的增加以及认知和语言能力的增长而发现它们。另一种并非唯一的可能性是，手势在掌握日益变化的交流需求的过程中发挥着重要作用，这在孩子们学习产生不同的交流类型时变得明显。因此，除了为说话行为所招募的感觉运动信息外，手势可能是与不同交际目的相关的说话的一个不可分割的资源。这种可能性为手势-言语整合模型提出了新的问题：手势的使用是否依赖于交际类型的经验，如果是这样，语用方面是如何融入到言语的产生中的？在我们提出的项目中，我们采用定性和定量分析以及计算评估方法，纵向调查了4岁和5岁学龄前儿童的手势和语言整合是如何发展的，以及语境（即交际类型和物理环境）如何与儿童在互动中使用的不同手势技术相关。我们项目的结果将为手势和语言整合的综合发展理论提供重要贡献，包括说话的认知和语用方面。

Gesture and speech form an integrated communicative system. This integration has been investigated for the purpose of communication as well as learning. Recent research attempting to explain and model this integration focus on how gesture influences speech, and how gesture -- as a special form of action -- forces the speaker to incorporate sensorimotor information into the representation. In these cognitive approaches, the fact that gesture is employed in various communicative genres and thus interacts with speech for different discursive purposes is disregarded. Yet types of communicative genres have been shown to bear different cognitive and communicative requirements on the speaker and hearer: For example, a narration requires different structural knowledge and verbal skills from a speaker than an explanation. While for competent speakers, the appearance of gestures in different communicative genres might occur fluent, for children learning to cope with the variety of discursive contexts, the role of gesture for thinking and communicating might be better observable: One possibility is that for particular genres, gestures are required resources for depicting strategies facilitating mutual understanding, and children discover them with increasing experience in interaction and with growing cognitive and linguistic competencies. Another -- not exclusive -- possibility is that gesture plays an important role on the way to master increasingly changing communicative requirements, which becomes visible when children learn to produce different communicative genres. Thus, in addition to sensorimotor information that is recruited for the act of speaking, gesture might be an integral resource of speaking in relation to different communicative purposes. This possibility raises new questions for models of gesture-speech integration: Is gesture used in dependence of the experience in communicative genre and if so how is the pragmatic aspect incorporated into the production of speech? In our proposed project, we apply qualitative and quantitative analyses as well as computational evaluation methods in investigating longitudinally how the integration of gesture and speech develops in 4 and 5 years old preschool children and how context (meaning the communicative genre and physical surroundings) relates to the different gestural techniques that children use in interaction. The results of our project will provide a crucial contribution towards a comprehensive developmental theory of gesture and speech integration involving cognitive and pragmatic aspects of speaking.