Investigating Multi-Modality Tutoring: Toward Computer-Based Written English Support for Deaf Children

该项目继续并扩展了pi先前的写作导师工作，旨在帮助聋哑学生获得书面英语，其中自然语言处理技术与对手语结构（包括视觉元素和语义）如何影响书面英语的理解相结合，以帮助识别英语写作错误并改善纠正反馈。在目前的提案中，pi计划研究将动画签名代理整合到他们早期研究中制定的辅导系统框架中。为此目的，他们将研究和开发包含使用视觉签名模式的辅导策略。虽然他们没有提议开发动画标识代理本身（这是其他人的工作），但他们对确定添加这样的代理如何影响系统的辅导能力很感兴趣。重要的问题包括捕捉辅导对话的结构，以及在对话中的文本模式和签名模式之间做出原则性选择。提供混合模式教程对话的策略也将被考虑，例如，一些辅导将被签署，但可能需要参考屏幕上的静态文本或文本交付的教程材料的情况。此外，计划指标建议进行一系列探索性和评价研究，以确定应纳入哪些模式以及如何提出这些模式。通过这个过程，项目项目负责人期望在项目的各个方面被集成到整个系统之前发现它们需要改进。直接的影响将在自然语言生成领域，特别是多模态生成领域。对如何在辅导环境中整合各种形式的信息的理解将得到加强。从长远来看，希望pi的写作导师的发展将对聋哑人的读写能力产生积极的影响。

This project continues and expands upon the PIs' prior work on a writing tutor designed to aid deaf students in acquiring written English, in which natural language processing techniques are combined with an understanding of how the structure of sign language (both its visual elements and its semantics) may affect written English to help identify English writing errors and to improve corrective feedback. In the current proposal, the PIs plan to investigate the integration of an animated signing agent into the tutoring system framework laid out in their earlier research. To this end, they will examine and develop tutoring strategies that incorporate the use of a visual signed modality. Although they are not proposing to develop the animated sign agent itself (that being the work of others), they are interested in determining how the addition of such an agent might affect the tutoring capabilities of the system. Important issues involve capturing the structure of tutoring dialogues, as well as making principled choices between the textual and signed modalities within a dialogue. Strategies for delivering a mixed-modal tutorial dialogue will also be considered, e.g. for situations where some of the tutoring will be signed but may need to refer to either static text on the screen or to textually delivered tutorial material. In addition, the PIs propose an ongoing series of exploratory and evaluation studies to determine which modalities should be incorporated and how they should be presented. Through this process, the PIs expect to discover which individual aspects of the project need improvement before they are integrated into the overall system. The immediate impact will be in the area of natural language generation and, in particular, multi-modal generation. Understanding of how to integrate information from various modalities in a tutoring context will be enhanced. In the long term, it is hoped that development of the PIs' writing tutor will positively impact the literacy skills of people who are deaf.