Getting Meta: A Multimodal Approach for Detecting Unsafe Conversations within Instagram Direct Messages of Youth
Getting Meta: A Multimodal Approach for Detecting Unsafe Conversations within Instagram Direct Messages of Youth
复制标题
DOI：
10.1145/3579608
复制
发表时间：
2023-04
影响因子：
--
通讯作者：
Shiza Ali;Afsaneh Razi;Seunghyun Kim;Ashwaq Alsoubai;Chen Ling;M. de Choudhury;P. Wisniewski;G. Stringhini
中科院分区：
文献类型：
--
作者：
Shiza Ali;Afsaneh Razi;Seunghyun Kim;Ashwaq Alsoubai;Chen Ling;M. de Choudhury;P. Wisniewski;G. Stringhini
Instagram是年轻人中最受欢迎的社交媒体平台之一，最近因可能对我们年轻一代的安全和福祉有害而受到审查。自动化的风险检测方法可能是帮助减轻其中一些风险的一种方法，如果这种算法既准确又与青少年在社交媒体平台上面临的在线伤害类型相关。然而，Instagram即将转向端到端加密私人对话将限制平台可用于检测和减轻此类风险的数据类型。在本文中，我们研究了哪些指标最有助于自动检测Instagram私人对话中的风险，并着眼于高级元数据，这些元数据在端到端加密的情况下仍然可用。为此，我们收集了172名年轻人（13-21岁）的Instagram数据，并要求他们识别让他们感到不舒服或不安全的私人消息对话。我们的参与者对28，725次对话进行了风险标记，其中包含4，181，970条直接消息，包括文本帖子和图像。基于这个丰富的多模态数据集，我们测试了多个特征集（元数据、语言线索和图像特征），并训练了分类器来检测有风险的对话。总的来说，我们发现元数据特征（例如，会话长度，参与者参与度的代理）是风险会话的最佳预测因子。然而，对于区分风险类型，不同的语言和媒体线索是最好的预测。根据我们的研究结果，我们为存在端到端加密的AI风险检测系统提供了设计意义。更广泛地说，我们的工作有助于青少年在线安全的文献走向更强大的解决方案，直接考虑到青少年的生活风险经验的风险检测。
Instagram, one of the most popular social media platforms among youth, has recently come under scrutiny for potentially being harmful to the safety and well-being of our younger generations. Automated approaches for risk detection may be one way to help mitigate some of these risks if such algorithms are both accurate and contextual to the types of online harms youth face on social media platforms. However, the imminent switch by Instagram to end-to-end encryption for private conversations will limit the type of data that will be available to the platform to detect and mitigate such risks. In this paper, we investigate which indicators are most helpful in automatically detecting risk in Instagram private conversations, with an eye on high-level metadata, which will still be available in the scenario of end-to-end encryption. Toward this end, we collected Instagram data from 172 youth (ages 13-21) and asked them to identify private message conversations that made them feel uncomfortable or unsafe. Our participants risk-flagged 28,725 conversations that contained 4,181,970 direct messages, including textual posts and images. Based on this rich and multimodal dataset, we tested multiple feature sets (metadata, linguistic cues, and image features) and trained classifiers to detect risky conversations. Overall, we found that the metadata features (e.g., conversation length, a proxy for participant engagement) were the best predictors of risky conversations. However, for distinguishing between risk types, the different linguistic and media cues were the best predictors. Based on our findings, we provide design implications for AI risk detection systems in the presence of end-to-end encryption. More broadly, our work contributes to the literature on adolescent online safety by moving toward more robust solutions for risk detection that directly takes into account the lived risk experiences of youth.