Learning representations of players' emotions and state for next generation gaming

电脑和游戏机游戏平台越来越多地配备了传感器，如摄像头和深度传感器，可以在玩家玩游戏时观察和跟踪他们。然而，实时分析这些数据仍然是一个科学和技术挑战，而不仅仅是身体追踪的一部分，在游戏过程中可靠地识别和跟踪用户的精确情绪状态。与蒙特利尔的合作伙伴育碧合作，我们建议利用机器学习技术的最新进展来实现这一目标。具体来说，我们将采用最新的方法在大数据集上训练深度神经网络，以学习玩家捕获的图像（尤其是面部）和情绪状态之间的映射。研究将集中于改进这些技术，使其能够快速和透明地适应每个球员的特定面部表情模式。由于这种技术，加拿大电脑游戏行业将获得主要的竞争优势，因为游戏系统能够实时准确地感知玩家的情绪状态，这将使游戏能够适应这种状态，并且有望成为打造下一代真正沉浸式游戏体验的关键。

Computer and console gaming platforms are increasingly equiped with sensors, such as cameras and depth sensors, that can watch and track players while they are playing a game. It however remains a scientific and technical challenge to analyse this data in real time, beyond mere parts of body tracking, to reliably recognize and track a user's precise emotional state during gameplay. With our Montreal-based partner Ubisoft, we propose to leverage recent advances in machine learning techniques to achieve this. Specifically we will employ recent approaches for training deep neural networks on big datasets, to learn a mapping between a player's captured images (especially the face) and emotional state. The research will focus on improving these techniques to allow fast and transparent adaptation to each player's specific modes of facial expression. The Canadian computer gaming industry would gain a major competitive advantage thanks to such technology, since the ability of a gaming system to accurately perceive a player's emotional state in real time will enable the game to adapt to it, and is anticipated to be the key for crafting the next generation of truly immersive gaming experience.