Challenges in CISE: Creating Conversational Agents for Language Training: Technologies for the Next Generation of Interactive Systems

9726363 Cole，Ronald Massaro，Dominic Oregon Graduate Institute of Science在CISE中的挑战：为下一代交互系统的语言培训技术创建对话代理这项研究解决的挑战是为语言培训领域，特别是听力障碍者的口语培训创建一个现实的会话代理。会话代理由四项关键技术组成，两项用于输出，两项用于输入：(1)精确移动发音机构(嘴唇、舌头等)的说话脸部的3D模型。(2)基于串联合成的自然文本到语音转换，它可以快速学习新的声音，并且能够进行用于在语言训练中提供反馈的那种超发音；(3)视觉语音识别，使用不显眼的桌面摄像头来辅助语音识别并提供有关学生发音的信息；以及(4)听觉语音识别，以便系统能够理解所说的话，并且使得系统能够检测发音错误。需要在语音识别和理解、语言生成和语音合成、面部线索和头部动作的识别以及人工说话头部的语音生成方面取得研究进展，以提高准确性并使技术适合一般会话代理，特别是语言培训。参与式设计实验将在波特兰地区的学校中进行。Tucker-Maxon口语学校将使用对话代理来教授听力受损的学生使用和理解听觉和视觉语音。这一现实的试验台将为研究团队提供关于该系统有用性的宝贵反馈。

9726363 Cole, Ronald Massaro, Dominic Oregon Graduate Institute of Science Challenges in CISE: Creating Conversational Agents for Language Training Technologies for the Next Generation of Interactive Systems The challenge addressed by this research is the creation of a realistic conversational agent for the domain of language training, and for spoken language training of the hearing impaired in particular. The conversational agent consists of four key technologies, two for output and two for input: (1) a 3D model of a talking face with accurate movement of the articulators (lips, tongue, etc.) and a variety of facial gestures and expression, (2) natural text-to-speech based on concatenative synthesis which can quickly learn new voices and is capable of hyperarticulation of the kind used to provide feedback in language training, (3) visual speech recognition using an unobtrusive desk-top camera to aid speech recognition and to provide information about the articulators of the student, and (4) auditory speech recognition so the system can understand what is said and also so the system can detect mispronunciations. Research advances are required in speech recognition and understanding, language generation and speech synthesis, recognition of facial cues and head movements, and generation of speech by artificial talking heads to improve the accuracy and to tailor the technology to conversational agents in general and language training in particular. Participatory design experiments will be conducted with schools in the Portland area. The Tucker-Maxon Oral School will use the conversational agents to teach hearing-impaired students to use and understanding auditory and visual speech. This realistic testbed will provide the research team with invaluable feedback on the usefulness of the system.