Learning representations of players' emotions and state for next generation gaming

计算机和游戏机游戏平台越来越多地配备了传感器，如摄像头和深度传感器，可以在玩家玩游戏时监视和跟踪他们。然而，实时分析这些数据，而不仅仅是身体跟踪的一部分，以可靠地识别和跟踪用户在玩游戏期间的准确情绪状态，仍然是一个科学和技术挑战。与总部位于蒙特利尔的合作伙伴育碧一起，我们建议利用机器学习技术的最新进展来实现这一目标。具体地说，我们将使用最近的方法在大数据集上训练深度神经网络，以学习玩家捕获的图像(特别是面部)和情绪状态之间的映射。这项研究将集中在改进这些技术，以允许快速和透明地适应每个玩家的特定面部表情模式。多亏了这种技术，加拿大的计算机游戏行业将获得重大的竞争优势，因为游戏系统实时准确感知玩家情绪状态的能力将使游戏能够适应它，预计这将是打造下一代真正身临其境的游戏体验的关键。

Computer and console gaming platforms are increasingly equiped with sensors, such as cameras and depth sensors, that can watch and track players while they are playing a game. It however remains a scientific and technical challenge to analyse this data in real time, beyond mere parts of body tracking, to reliably recognize and track a user's precise emotional state during gameplay. With our Montreal-based partner Ubisoft, we propose to leverage recent advances in machine learning techniques to achieve this. Specifically we will employ recent approaches for training deep neural networks on big datasets, to learn a mapping between a player's captured images (especially the face) and emotional state. The research will focus on improving these techniques to allow fast and transparent adaptation to each player's specific modes of facial expression. The Canadian computer gaming industry would gain a major competitive advantage thanks to such technology, since the ability of a gaming system to accurately perceive a player's emotional state in real time will enable the game to adapt to it, and is anticipated to be the key for crafting the next generation of truly immersive gaming experience.