Intelligent Multimedia Entertainment: Online monitoring of affective and cognitive states

娱乐产业正面临着电子平台的多样化、人们对游戏和数字媒体消费的快速增长、兴趣和情绪的波动等重大变化。为了应对这些挑战，媒体娱乐行业现在必须开发下一代自适应系统；也就是说，为了定制内容并提供最积极的以用户为中心的体验，他们必须包括基于生物识别和行为标记的娱乐人员认知和情感状态的在线监测。在确定可靠的标记并将其纳入媒体的互动性方面，需要进行关键的创新。我们的方法结合了实验心理学和认知人体工程学来研究生理和行为指标，作为工作量、信息寻找模式、参与度和娱乐人员与Liveshout社交平台互动的情绪的在线标记。首先，拟议的项目将允许基于心电图（例如，心率）、眼球追踪（例如，注视时间和瞳孔扩张）和面部阅读（例如，喜悦和悲伤）特征的整合，初步开发艺人的功能状态（EFS）。其次，该项目旨在通过一个架构的概念化来促进基于EFS的交互式娱乐的发展，该架构能够将Liveshout平台的内容实时地适配到EFS中。该项目的结果将为设计适应性和互动性的社会媒体确定指导方针和建议，并将对加拿大的技术、金融和教育部门产生积极影响。该项目的成果还将对智能教育、交通和公共安全等其他领域产生有益影响。

The entertainment industry is faced with major changes such as the diversification of electronic platforms as well as the rapid increase of people's consumption of games and digital media, and their fluctuating interests and moods. To address these challenges, media entertainment industries must now develop next-generation adaptive systems; that is, in order to tailor content and provide the most positive user-centered experience, they must include online monitoring of entertainees' cognitive and affective state based on biometric and behavioral markers. Key innovations are needed in terms of identifying reliable markers and their integration within the interactivity of the media. Our approach combines experimental psychology and cognitive ergonomics to investigate physiological and behavioural metrics as online markers of workload, information seeking patterns, engagement, and mood of the entertainee interacting with the Liveshout social platform. First, the proposed project will allow the initial development of an entertainee's functional state (EFS) based on the integration of electrocardiography (e.g., heart rate), eye-tracking (e.g., fixation duration and pupil dilation) and face-reading (e.g., joy and sadness) features. Second, the project aims to contribute to the development of IAS-based interactive entertainment through the conceptualization of an architecture that enables the real-time adaptation of the Liveshout platform content to the EFS. Results of this proposed project will define guidelines and recommendations for designing adaptive and interactive social media, and will positively impact the Canadian technological, financial and education sectors. The outcomes of the project will also have beneficial impacts on other areas such as intelligent tutoring, transportation and public security.