Challenges in CISE: Creating Conversational Agents for Language Training: Technologies for the Next Generation of Interactive Systems

926363 Cole, Ronald Massaro， Dominic俄勒冈科学研究所CISE的挑战：为下一代交互系统的语言训练技术创建会话代理本研究解决的挑战是为语言训练领域创建一个现实的会话代理，特别是为听力受损者的口语训练。会话代理包括四项关键技术，两项用于输出，两项用于输入：(1)具有发音器官（嘴唇、舌头等）的精确运动和各种面部手势和表情的说话面部3D模型；(2)基于连接合成的自然文本到语音，可以快速学习新的声音，并能够在语言训练中提供反馈的那种超发音；(3)使用不显眼的桌面摄像头进行视觉语音识别，以帮助语音识别并提供有关学生发音器官的信息。(4)听觉语音识别，这样系统就可以理解所说的内容，也可以检测错误的发音。需要在语音识别和理解、语言生成和语音合成、面部线索和头部运动识别、人工说话头生成语音等方面取得研究进展，以提高准确性，并使技术适应一般的会话代理，特别是语言训练。参与式设计实验将在波特兰地区的学校进行。塔克-马克森口语学校将使用会话代理来教听力受损的学生使用和理解听觉和视觉语言。这个现实的测试平台将为研究团队提供关于系统有用性的宝贵反馈。

9726363 Cole, Ronald Massaro, Dominic Oregon Graduate Institute of Science Challenges in CISE: Creating Conversational Agents for Language Training Technologies for the Next Generation of Interactive Systems The challenge addressed by this research is the creation of a realistic conversational agent for the domain of language training, and for spoken language training of the hearing impaired in particular. The conversational agent consists of four key technologies, two for output and two for input: (1) a 3D model of a talking face with accurate movement of the articulators (lips, tongue, etc.) and a variety of facial gestures and expression, (2) natural text-to-speech based on concatenative synthesis which can quickly learn new voices and is capable of hyperarticulation of the kind used to provide feedback in language training, (3) visual speech recognition using an unobtrusive desk-top camera to aid speech recognition and to provide information about the articulators of the student, and (4) auditory speech recognition so the system can understand what is said and also so the system can detect mispronunciations. Research advances are required in speech recognition and understanding, language generation and speech synthesis, recognition of facial cues and head movements, and generation of speech by artificial talking heads to improve the accuracy and to tailor the technology to conversational agents in general and language training in particular. Participatory design experiments will be conducted with schools in the Portland area. The Tucker-Maxon Oral School will use the conversational agents to teach hearing-impaired students to use and understanding auditory and visual speech. This realistic testbed will provide the research team with invaluable feedback on the usefulness of the system.