Investigating Multi-Modality Tutoring: Toward Computer-Based Written English Support for Deaf Children

点击翻译按钮获取中文摘要

This project continues and expands upon the PIs' prior work on a writing tutor designed to aid deaf students in acquiring written English, in which natural language processing techniques are combined with an understanding of how the structure of sign language (both its visual elements and its semantics) may affect written English to help identify English writing errors and to improve corrective feedback. In the current proposal, the PIs plan to investigate the integration of an animated signing agent into the tutoring system framework laid out in their earlier research. To this end, they will examine and develop tutoring strategies that incorporate the use of a visual signed modality. Although they are not proposing to develop the animated sign agent itself (that being the work of others), they are interested in determining how the addition of such an agent might affect the tutoring capabilities of the system. Important issues involve capturing the structure of tutoring dialogues, as well as making principled choices between the textual and signed modalities within a dialogue. Strategies for delivering a mixed-modal tutorial dialogue will also be considered, e.g. for situations where some of the tutoring will be signed but may need to refer to either static text on the screen or to textually delivered tutorial material. In addition, the PIs propose an ongoing series of exploratory and evaluation studies to determine which modalities should be incorporated and how they should be presented. Through this process, the PIs expect to discover which individual aspects of the project need improvement before they are integrated into the overall system. The immediate impact will be in the area of natural language generation and, in particular, multi-modal generation. Understanding of how to integrate information from various modalities in a tutoring context will be enhanced. In the long term, it is hoped that development of the PIs' writing tutor will positively impact the literacy skills of people who are deaf.