Fusion of Face and Speech Characteristics for Real-Time Recognition of Behaviors in Human-Centric E-Learning

在过去的几十年里，已经采用了几项新技术来增强在线电子学习计划。NVS学习公司寻求提供基于虚拟和沉浸式环境的交互式培训计划，这些环境可以根据有效的领导、管理、客户服务、销售等方面的需要发展积极的行为技能。尽管仍未得到充分利用，但最新的网络摄像头和麦克风技术以及信号处理和模式识别技术将允许与此类环境进行更自然和不引人注目的交互。然而，利用这些技术的商业系统仅限于对人脸和语音信息进行离线后分析，以及识别一小部分表情。基于视频捕获的人脸和语音的行为技能实时识别是一个具有挑战性的大规模电子学习应用。 这个项目研究了最先进的技术，适用于基于人脸和语音模式的电子学习中行为技能的实时识别。给定在与电子学习环境交互期间捕捉学习者面部和语音变化的视频流，需要自动化系统来学习和解释他/她的行为技能，例如学习者的言语言语流，并向学习者提供个性化的反馈。将对几种先进的信号处理和模式技术进行回顾和比较。特别是，将考虑基于补丁的稀疏表示、多实例学习和集成学习技术来设计行为检测器池。该项目旨在为NVS学习提供所需的专业知识，为客户应用程序部署强大的交互式电子学习系统。将使用RECOLA公共数据集的视频对有前景的技术进行基准测试，并将从准确性和资源要求方面评估性能。

Over the past few decades, several new technologies have been adopted to enhance online e-learning programs. NVS Learning Inc. seeks to deliver interactive training programs based on virtual and immersive environments that develop positive behavioral skills, as needed in effective leadership, management, customer service, sales, etc. Although still underexploited, recent developments webcam and microphone technologies, as well as signal processing and pattern recognition techniques would allow for more natural and unobtrusive interactions with such environments. However, commercial systems that exploit these technologies are limited to offline post-analysis of face and speech information and recognition of a small set of expressions. Real-time recognition of behavioral skills based on faces and speech captured in videos is a challenging large-scale e-learning application. This project investigates state-of-the-art techniques that are suitable for the real-time recognition of behavioral skills in e-learning based on face and speech modalities. Given a video stream that captures the variations in the learner's face and speech during interactions with an e-learning environment, an automated system is required to learn and interpret his/her behavioral skills, e.g., the flow of the learner's speech verbalizations, and provide the learner with personalised feedback. Several advanced signal processing and pattern techniques will be reviewed and compared. In particular, patch-based sparse representation, multiple instance learning, and ensemble learning techniques will be considered to design a pool of behavior detectors. This project aims to provide NVS Learning with the expertise required to deploy robust interactive e-learning systems for customer applications. Benchmarking of promising techniques will be performed with videos from the RECOLA public dataset, and performance will be assessed in terms of accuracy and resource requirements.