Challenges in CISE: Creating Conversational Agents for Language Training: Technologies for the Next Generation of Interactive Systems

9726363 Cole，Ronald Massaro，多米尼克俄勒冈科学研究所 CISE 中的挑战：为下一代交互系统的语言训练技术创建​​对话代理 这项研究解决的挑战是为语言训练领域，特别是听力受损者的口语训练创建一个现实的对话代理。 对话代理由四项关键技术组成，两项用于输出，两项用于输入：(1) 说话人脸的 3D 模型，具有发音器官（嘴唇、舌头等）的精确运动以及各种面部姿势和表情，(2) 基于连接合成的自然文本到语音，可以快速学习新声音，并能够进行用于在语言训练中提供反馈的超清晰表达，(3) 使用不显眼的视觉语音识别桌面摄像头帮助语音识别并提供有关学生发音器官的信息，以及（4）听觉语音识别，以便系统可以理解所说的内容，并且系统可以检测错误发音。 需要在语音识别和理解、语言生成和语音合成、面部线索和头部运动识别以及人工说话头生成语音方面取得研究进展，以提高准确性并根据一般会话代理和特别是语言训练定制技术。 参与式设计实验将在波特兰地区的学校中进行。 塔克-马克森口语学校将使用对话代理来教听力受损的学生使用和理解听觉和视觉言语。 这个现实的测试平台将为研究团队提供有关系统实用性的宝贵反馈。

9726363 Cole, Ronald Massaro, Dominic Oregon Graduate Institute of Science Challenges in CISE: Creating Conversational Agents for Language Training Technologies for the Next Generation of Interactive Systems The challenge addressed by this research is the creation of a realistic conversational agent for the domain of language training, and for spoken language training of the hearing impaired in particular. The conversational agent consists of four key technologies, two for output and two for input: (1) a 3D model of a talking face with accurate movement of the articulators (lips, tongue, etc.) and a variety of facial gestures and expression, (2) natural text-to-speech based on concatenative synthesis which can quickly learn new voices and is capable of hyperarticulation of the kind used to provide feedback in language training, (3) visual speech recognition using an unobtrusive desk-top camera to aid speech recognition and to provide information about the articulators of the student, and (4) auditory speech recognition so the system can understand what is said and also so the system can detect mispronunciations. Research advances are required in speech recognition and understanding, language generation and speech synthesis, recognition of facial cues and head movements, and generation of speech by artificial talking heads to improve the accuracy and to tailor the technology to conversational agents in general and language training in particular. Participatory design experiments will be conducted with schools in the Portland area. The Tucker-Maxon Oral School will use the conversational agents to teach hearing-impaired students to use and understanding auditory and visual speech. This realistic testbed will provide the research team with invaluable feedback on the usefulness of the system.