Fusion of Face and Speech Characteristics for Real-Time Recognition of Behaviors in Human-Centric E-Learning

点击翻译按钮获取中文摘要

Over the past few decades, several new technologies have been adopted to enhance online e-learning programs. NVS Learning Inc. seeks to deliver interactive training programs based on virtual and immersive environments that develop positive behavioral skills, as needed in effective leadership, management, customer service, sales, etc. Although still underexploited, recent developments webcam and microphone technologies, as well as signal processing and pattern recognition techniques would allow for more natural and unobtrusive interactions with such environments. However, commercial systems that exploit these technologies are limited to offline post-analysis of face and speech information and recognition of a small set of expressions. Real-time recognition of behavioral skills based on faces and speech captured in videos is a challenging large-scale e-learning application. This project investigates state-of-the-art techniques that are suitable for the real-time recognition of behavioral skills in e-learning based on face and speech modalities. Given a video stream that captures the variations in the learner's face and speech during interactions with an e-learning environment, an automated system is required to learn and interpret his/her behavioral skills, e.g., the flow of the learner's speech verbalizations, and provide the learner with personalised feedback. Several advanced signal processing and pattern techniques will be reviewed and compared. In particular, patch-based sparse representation, multiple instance learning, and ensemble learning techniques will be considered to design a pool of behavior detectors. This project aims to provide NVS Learning with the expertise required to deploy robust interactive e-learning systems for customer applications. Benchmarking of promising techniques will be performed with videos from the RECOLA public dataset, and performance will be assessed in terms of accuracy and resource requirements.