ITR: Creating the Next Generation of Intelligent Animated Conversational Agents

这是一项为期五年的持续奖励的第一年资助。该项目的目标是通过设计基于计算机的互动式阅读导师，结合新的语音和语言技术，提高有阅读问题的儿童的阅读成绩。阅读导师将帮助英语和西班牙语儿童学习阅读，为课堂教师和阅读专家提供工具，指导和锻炼阅读所需的听觉、视觉和语言技能，包括语音辨别、语音生成、语音意识、语音到字母映射、词汇、流利度和理解。这些导师将与阅读专家和教学设计师合作设计、测试和完善，并在科罗拉多州博尔德市小学特殊教育项目的儿童中进行测试。导师将结合新的和改进的听觉和视觉语音识别和面部动画技术。俄勒冈研究生院（OGI）、美洲普埃布拉大学（UDLA）、加州大学圣克鲁斯分校（UCSC）、加州大学圣地亚哥分校（UCSD）和科罗拉多大学（CU）这五个合作站点将开发语音和语言技术。儿童语音识别器的研究和开发将在西班牙语的UDLA和英语的OGI进行。此外，这些网站将设计和开发语音语料库，以进行识别研究。加州大学圣地亚哥分校将进行研究，开发头部跟踪和语音阅读系统，并设计和开发视频语料库，以实现这项研究。UCSC将进行研究，以开发具有改进动画能力的新动画面孔。系统集成将在OGI进行，它将把听觉和视觉识别系统以及面部动画系统集成到CSLU工具包中。科罗拉多大学将与教师、教学设计师和学生合作开发英语阅读导师，并对项目成果进行评估。UDLA还将开发和测试导师的西班牙语版本。该项目预计将在听觉和视觉识别技术方面取得重大进展，包括对儿童语音的准确识别，对语音视觉特征的准确识别，以及在语言训练应用中首次将听觉和视觉语音识别实时集成。此外，PI和他的团队将对儿童语言结构的理解以及阅读中听觉和视觉信息的处理达到一个新的水平。面部动画有望在吸引儿童方面发挥重要作用，使他们能够更多地享受学习体验，从而将更多时间花在任务上。PI希望证明使用可见发音器的面部动画将提高语音识别和语音生成技能，提高语音意识和阅读能力。通过将听觉和视觉语音识别以及语音生成技术整合到动画代理中，并设计将这些代理整合到精心设计的阅读程序中的阅读导师，PI希望提高学校的阅读成绩。为了优化这一成果，该项目正在与阅读专家密切合作，吸收他们的经验和最佳做法；制定形成性和总结性评估计划，确保对计划干预的结果进行公正和准确的评估。

This is the first year funding of a five-year continuing award. The goal of this project is to improve reading achievement of children with reading problems by designing computer-based interactive reading tutors that incorporate new speech and language technologies. The reading tutors will help English- and Spanish-speaking children learn to read by providing classroom teachers and reading specialists with tools to instruct and exercise the set of auditory, visual and linguistic skills needed to read, speech discrimination, speech production, phonological awareness, sound-to-letter mappings, vocabulary, fluency and comprehension. The tutors will be designed, tested and refined in collaboration with reading specialists and instructional designers, and tested with children in special education programs in elementary schools in Boulder Colorado. The tutors will incorporate new and improved auditory and visual speech recognition and facial animation technologies. Five partner sites - Oregon Graduate Institute (OGI), Universidad de las Americas, Puebla (UDLA), University of California, Santa Cruz (UCSC), University of California, San Diego (UCSD) and the University of Colorado (CU), will develop speech and language technologies. Research and development of children's speech recognizers will be conducted at UDLA for Spanish and at OGI for English. In addition, these sites will design and develop speech corpora to enable recognition research. UCSD will conduct research leading to development of head tracking and speech reading systems, and design and develop video corpora to enable this research. UCSC will conduct research leading to development of new animated faces with improved animation capabilities. System integration will be conducted at OGI, which will integrate auditory and visual recognition systems and facial animation systems into the CSLU Toolkit. CU will develop English reading tutors in collaboration with teachers, instructional designers and students, and conduct evaluations of project outcomes. UDLA will also develop and test Spanish versions of the tutors.The project is expected to produce significant advances in auditory and visual recognition technologies, including accurate recognition of children's speech, accurate recognition of visual features of speech, and the first real-time integration of auditory and visual speech recognition in language training applications. In addition, the PI and his team will achieve a new level of understanding of the structure of children's speech, and the processing of auditory and visual information in reading. Facial animation is expected to play a major role in engaging children, enabling them to enjoy the learning experience more and therefore spend more time on task. The PI expects to demonstrate that facial animation using visible articulators will improve speech discrimination and speech production skills, improved phonological awareness and improved reading. By integrating auditory and visual speech recognition and speech generation technologies into animated agents, and designing reading tutors that incorporate these agents in a well designed reading program, the PI hopes to improve reading achievement in schools. To optimize this outcome, the PI is working closely with reading specialists to incorporate their experience and best practices; and by developing formative and summative evaluation plans that assure fair and accurate assessment of the outcomes of the planned interventions.