Quantifying the Relationships between Everyday Objects and Emotional States through Deep Learning Based Image Analysis Using Smartphones
Quantifying the Relationships between Everyday Objects and Emotional States through Deep Learning Based Image Analysis Using Smartphones
复制标题
DOI：
10.1145/3380997
复制
发表时间：
2020-03-01
期刊：
影响因子：
--
通讯作者：
Musolesi, Mirco
中科院分区：
文献类型：
--
作者：
Darvariu, Victor-Alexandru;Convertino, Laura;Musolesi, Mirco
人们越来越关注使用传感器和用户生成的信息来推断个体的情绪状态的问题，这些信息与GPS轨迹、社交媒体数据和智能手机交互模式一样多样。一个很少受到关注的方面是从个人周围环境中提取的视觉上下文信息的使用以及它们如何与之相关。在本文中，我们提出了一个观察性研究，研究使用深度学习技术从智能手机图像中自动提取的个人和视觉环境中存在的对象之间的关系。我们开发了MyMood，这是一款智能手机应用程序，允许用户定期记录他们的情绪状态以及日常生活中的照片，同时被动收集传感器测量结果。我们对22名参与者进行了一项野外研究，收集了3,305份带有照片的情绪报告。我们的研究结果表明，个人周围的物体和自我报告的情绪状态强度之间的关联依赖于上下文。这项工作的应用可能有很多，从室内和室外空间的设计到积极行为干预的智能应用程序的开发，以及更广泛的支持计算心理学研究。
There has been an increasing interest in the problem of inferring emotional states of individuals using sensor and user-generated information as diverse as GPS traces, social media data and smartphone interaction patterns. One aspect that has received little attention is the use of visual context information extracted from the surroundings of individuals and how they relate to it. In this paper, we present an observational study of the relationships between the emotional states of individuals and objects present in their visual environment automatically extracted from smartphone images using deep learning techniques. We developed MyMood, a smartphone application that allows users to periodically log their emotional state together with pictures from their everyday lives, while passively gathering sensor measurements. We conducted an in-the-wild study with 22 participants and collected 3,305 mood reports with photos. Our findings show context-dependent associations between objects surrounding individuals and self-reported emotional state intensities. The applications of this work are potentially many, from the design of interior and outdoor spaces to the development of intelligent applications for positive behavioral intervention, and more generally for supporting computational psychology studies.