An AI-based Visualisation Feedback System for Speech Training
An AI-based Visualisation Feedback System for Speech Training
批准号:
2717184
负责人:
金额:
$0.0万
依托单位:
依托单位国家:
英国
项目类别:
Studentship
财政年份:
2022
资助国家:
英国
项目状态:
未结题
起止时间:
2022 至 --
中文摘要
向语言学习者提供有效的个性化反馈,使他们能够控制自己的学习体验,并允许他们按照自己的节奏进行练习。然而,在语言学习领域,学习者往往需要额外的培训或人工干预才能解释反馈。其中一个目标是为公开演讲培训提供详细的实时反馈。因此,该系统将重点放在他们应该在多大程度上模仿母语人士或做自己,以吸引观众。稍后,我们将调整系统以支持有特殊教育需要的学习者,特别是那些难以传达情感的学习者。语音中的情感分类是一项具有挑战性的任务,因为情感通常只在句子之间发生微妙的变化,而且很难将句子归类为单独的情感。为了扩展现有的系统,强化学习技术将被用于情感识别和反馈,并允许专家手动纠正模型给出的自动反馈,并更新代理的策略。此外,我们将使用最新的图神经网络(GNN)来对情感进行分类,这些情感可以使用一种自我监督的方法进行训练,在这种方法中,没有标签或有限的标签。语音段之间的关系可以用图形来表示,并且可以用于会话情感分析。将考虑进一步操纵演讲者的音频,以不仅调整情绪基调,而且调整音调、口音和发音。此外,还将研究语音中的置信度和不确定性的检测以及语音和手势分析的结合。还将提供反馈意见,以提高学习者的写作技能。总之,已经提出了一种方法,其中公开演讲训练第一次由在视觉仪表板上提供的详细反馈来支持,该反馈不仅包括转录和音调信息,而且还包括情感信息。学习者可以使用转录来识别发音问题,并查看他们的音调和情绪在整个演讲过程中是如何变化的,以更有效地提高他们的口语技能。情感分类将使用最先进的强化学习网络和GNN来提供,它们结合了来自多个专家的人工反馈,以实现高分类精度。
英文摘要
Effective individualised feedback provided to language learners enables them to control their learning experience and allows them to practice at their own pace. However, in the language learning domain, learners often require additional training or intervention by humans to be able to interpret the feedback. One objective is to provide detailed real-time feedback for public speaking training purposes. The system will therefore focus on to what extent they should imitate native speakers or be themselves in order to engage the audience. Later we will adapt the system to support learners with special educational needs, especially those with difficulties conveying emotion.Emotion classification in speech is a challenging task as emotion usually only changes subtly between sentences, and it is difficult to classify sentences as individual emotions. To expand the current system, reinforcement learning techniques will be used for emotion recognition and feedback and allow experts to manually correct the automatic feedback given by the model and to update the agent's policy. Moreover, we will use state of the art graph neural networks (GNNs) to classify emotions which can be trained using a self-supervised method in which there is no labelling or limited labelling. Relationships between speech segments can be represented using graphs and can be used for conversational emotion analysis. Further manipulation of the speakers' audio to adapt not only the emotional tone but also pitch accent and pronunciation will be considered. Furthermore, the detection of confidence and uncertainty in speech and the combination of speech and gesture analysis will be studied. Feedback will also be provided to improve the learners writing skills. In Summary, a method has been proposed where for the first time, public speaking training is supported by detailed feedback provided on a visual dashboard including not only the transcription and pitch information, but also emotion information. Learners can use transcription to identify pronunciation issues and view how their pitch and emotion vary throughout their speech to more effectively improve their speaking skills. Emotion classification will be provided using state of the art reinforcement learning networks and GNNs which incorporate manual feedback from multiple experts to achieve a high classification accuracy.
期刊论文(0)
专著(0)
科研奖励(0)
会议论文
国内基金
海外基金
登录
查看更多内容
Data-driven Recommendation System Construction of an Online Medical Platform Based on the Fusion of Information
-
批准号:--
-
项目类别:外国青年学者研究基金项目
-
资助金额:--
-
批准年份:2024
-
负责人:江洋子
-
依托单位:
Incentive and governance schenism study of corporate green washing behavior in China: Based on an integiated view of econfiguration of environmental authority and decoupling logic
-
批准号:--
-
项目类别:外国学者研究基金项目
-
资助金额:--
-
批准年份:2024
-
负责人:YU BYUNGJUN
-
依托单位:
Exploring the Intrinsic Mechanisms of CEO Turnover and Market Reaction: An Explanation Based on Information Asymmetry
-
批准号:W2433169
-
项目类别:外国学者研究基金项目
-
资助金额:--
-
批准年份:2024
-
负责人:HAOFEI ZHANG
-
依托单位:
含Re、Ru先进镍基单晶高温合金中TCP相成核—生长机理的原位动态研究
-
批准号:52301178
-
项目类别:青年科学基金项目
-
资助金额:30.00万元
-
批准年份:2023
-
负责人:夏万顺
-
依托单位:
NbZrTi基多主元合金中化学不均匀性对辐照行为的影响研究
-
批准号:12305290
-
项目类别:青年科学基金项目
-
资助金额:30.00万元
-
批准年份:2023
-
负责人:苏钲雄
-
依托单位:
眼表菌群影响糖尿病患者干眼发生的人群流行病学研究
-
批准号:82371110
-
项目类别:面上项目
-
资助金额:49.00万元
-
批准年份:2023
-
负责人:邹海东
-
依托单位:
CuAgSe基热电材料的结构特性与构效关系研究
-
批准号:22375214
-
项目类别:面上项目
-
资助金额:50.00万元
-
批准年份:2023
-
负责人:周钲洋
-
依托单位:
镍基UNS N10003合金辐照位错环演化机制及其对力学性能的影响研究
-
批准号:12375280
-
项目类别:面上项目
-
资助金额:53.00万元
-
批准年份:2023
-
负责人:黄鹤飞
-
依托单位:
A study on prototype flexible multifunctional graphene foam-based sensing grid (柔性多功能石墨烯泡沫传感网格原型研究)
-
批准号:--
-
项目类别:--
-
资助金额:20万元
-
批准年份:2020
-
负责人:SAGAR RIZWAN UR REHMAN
-
依托单位:
基于大数据定量研究城市化对中国季节性流感传播的影响及其机理
-
批准号:82003509
-
项目类别:青年科学基金项目
-
资助金额:24.0万元
-
批准年份:2020
-
负责人:雷浩
-
依托单位: