An AI-based Visualisation Feedback System for Speech Training

基于人工智能的语音训练可视化反馈系统

• 批准号: 2717184
• 金额: --
• 依托单位: Durham University
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2022
• 资助国家: 英国
• 起止时间: 2022 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2717184
• 关键词: AI; based; Visualisation; Feedback; System

Effective individualised feedback provided to language learners enables them to control their learning experience and allows them to practice at their own pace. However, in the language learning domain, learners often require additional training or intervention by humans to be able to interpret the feedback. One objective is to provide detailed real-time feedback for public speaking training purposes. The system will therefore focus on to what extent they should imitate native speakers or be themselves in order to engage the audience. Later we will adapt the system to support learners with special educational needs, especially those with difficulties conveying emotion.Emotion classification in speech is a challenging task as emotion usually only changes subtly between sentences, and it is difficult to classify sentences as individual emotions. To expand the current system, reinforcement learning techniques will be used for emotion recognition and feedback and allow experts to manually correct the automatic feedback given by the model and to update the agent's policy. Moreover, we will use state of the art graph neural networks (GNNs) to classify emotions which can be trained using a self-supervised method in which there is no labelling or limited labelling. Relationships between speech segments can be represented using graphs and can be used for conversational emotion analysis. Further manipulation of the speakers' audio to adapt not only the emotional tone but also pitch accent and pronunciation will be considered. Furthermore, the detection of confidence and uncertainty in speech and the combination of speech and gesture analysis will be studied. Feedback will also be provided to improve the learners writing skills. In Summary, a method has been proposed where for the first time, public speaking training is supported by detailed feedback provided on a visual dashboard including not only the transcription and pitch information, but also emotion information. Learners can use transcription to identify pronunciation issues and view how their pitch and emotion vary throughout their speech to more effectively improve their speaking skills. Emotion classification will be provided using state of the art reinforcement learning networks and GNNs which incorporate manual feedback from multiple experts to achieve a high classification accuracy.

为语言学习者提供有效的个性化反馈，使他们能够控制自己的学习体验，并允许他们按照自己的节奏练习。然而，在语言学习领域，学习者通常需要人类的额外训练或干预才能解释反馈。其中一个目标是为公众演讲培训提供详细的实时反馈。因此，该系统将重点关注他们应该在多大程度上模仿母语人士或做自己，以吸引观众。稍后，我们将调整该系统，以支持有特殊教育需要的学习者，特别是那些在表达情感方面有困难的学习者。语音中的情绪分类是一项具有挑战性的任务，因为情绪在句子之间的变化通常很微妙，很难将句子分类为单个情绪。为了扩展当前的系统，强化学习技术将用于情感识别和反馈，并允许专家手动纠正模型给出的自动反馈并更新代理的策略。此外，我们将使用最先进的图形神经网络（gnn）对情绪进行分类，这些情绪可以使用无标签或有限标签的自监督方法进行训练。语音片段之间的关系可以用图形表示，并可用于会话情感分析。将考虑进一步操纵说话者的音频，以适应不仅是情感语气，而且音高重音和发音。在此基础上，研究了语音中的置信度和不确定性检测以及语音和手势分析的结合。我们也会提供反馈，以提高学习者的写作技巧。综上所述，本文首次提出了一种方法，即在可视化仪表板上提供详细的反馈，不仅包括转录和音高信息，还包括情绪信息。学习者可以使用抄写来识别发音问题，并观察他们的音高和情绪在整个演讲过程中的变化，从而更有效地提高他们的口语技能。情感分类将使用最先进的强化学习网络和gnn，其中包含来自多个专家的手动反馈，以实现高分类精度。