An AI-based Visualisation Feedback System for Speech Training

为语言学习者提供有效的个性化反馈，使他们能够控制自己的学习体验，并允许他们按照自己的节奏进行练习。然而，在语言学习领域，学习者通常需要额外的训练或人类干预才能解释反馈。一个目标是为公开演讲培训提供详细的实时反馈。因此，该系统将侧重于他们应该在多大程度上模仿母语人士或做他们自己，以吸引观众。之后，我们将调整系统，以支持有特殊教育需求的学习者，特别是那些难以表达情感的学习者。语音中的情感分类是一项具有挑战性的任务，因为情感通常只在句子之间发生微妙变化，很难将句子分类为单个情感。为了扩展当前的系统，强化学习技术将用于情感识别和反馈，并允许专家手动纠正模型给出的自动反馈，并更新代理的策略。此外，我们将使用最先进的图神经网络（GNN）来对情绪进行分类，这些情绪可以使用无标签或有限标签的自监督方法进行训练。语音片段之间的关系可以使用图形来表示，并且可以用于会话情感分析。将考虑进一步操纵扬声器的音频，以不仅适应情感音调，而且适应音高、重音和发音。此外，将研究语音中的置信度和不确定性的检测以及语音和手势分析的结合。此外，还将提供反馈，以提高学习者的写作技巧。总之，已经提出了一种方法，其中公开演讲训练第一次由在视觉仪表板上提供的详细反馈支持，所述详细反馈不仅包括转录和音高信息，而且还包括情绪信息。学习者可以使用转录来识别发音问题，并查看他们的音高和情绪如何在整个演讲中变化，以更有效地提高他们的口语技能。情感分类将使用最先进的强化学习网络和GNN提供，这些网络结合了来自多个专家的手动反馈，以实现高分类准确度。

Effective individualised feedback provided to language learners enables them to control their learning experience and allows them to practice at their own pace. However, in the language learning domain, learners often require additional training or intervention by humans to be able to interpret the feedback. One objective is to provide detailed real-time feedback for public speaking training purposes. The system will therefore focus on to what extent they should imitate native speakers or be themselves in order to engage the audience. Later we will adapt the system to support learners with special educational needs, especially those with difficulties conveying emotion.Emotion classification in speech is a challenging task as emotion usually only changes subtly between sentences, and it is difficult to classify sentences as individual emotions. To expand the current system, reinforcement learning techniques will be used for emotion recognition and feedback and allow experts to manually correct the automatic feedback given by the model and to update the agent's policy. Moreover, we will use state of the art graph neural networks (GNNs) to classify emotions which can be trained using a self-supervised method in which there is no labelling or limited labelling. Relationships between speech segments can be represented using graphs and can be used for conversational emotion analysis. Further manipulation of the speakers' audio to adapt not only the emotional tone but also pitch accent and pronunciation will be considered. Furthermore, the detection of confidence and uncertainty in speech and the combination of speech and gesture analysis will be studied. Feedback will also be provided to improve the learners writing skills. In Summary, a method has been proposed where for the first time, public speaking training is supported by detailed feedback provided on a visual dashboard including not only the transcription and pitch information, but also emotion information. Learners can use transcription to identify pronunciation issues and view how their pitch and emotion vary throughout their speech to more effectively improve their speaking skills. Emotion classification will be provided using state of the art reinforcement learning networks and GNNs which incorporate manual feedback from multiple experts to achieve a high classification accuracy.