Understanding and generation of referring expressions using gaze

我们构建了日语指称表达语料库，为研究指称表达理解和生成的计算模型提供了基础数据。在我们的实验环境中，两个对话参与者协作解决几何难题。在每一次对话中，与对话演讲同步记录参与者的拼图排列、拼图操作和眼睛凝视。从转录的语音文本中提取指称表达，并用其指称对象和属性进行标注。所构建的语料库共包括131个对话。使用该语料库，基于机器学习框架构建了指代表达式理解和生成的计算模型。使用所构建的语料库对模型进行了评估，以显示模型的有效性。

We constructed a Japanese referring expression corpus, which serves as fundamental data for research on computational model of understanding and generation of referring expressions. In our experimental setting, two dialogue participants collaboratively solve geometrical puzzles. In each dialogue, arrangement of puzzle pieces, operations on puzzle pieces and eye gaze of the participants were recorded in synchronization with dialogue speech. Referring expressions were extracted from transcribed speech texts and annotated with their referents and attributes. The constructed corpus includes 131 dialogues in total. Using this corpus, computational models for understanding and generation of referring expressions were constructed based on a machine learning framework. The models were evaluated by using the constructed corpus to show the effectiveness of the models.