Collaborative Research: SaTC: CORE: Small: Understanding how visual features of misinformation influence credibility perceptions

如今的错误信息帖子越来越多地以视觉格式呈现，如图像、表情包和视频。与文本相比，视频处理速度更快，记忆力更强，更容易在社交媒体上分享。随着技术的发展，大众可以获得图像和视频操作，视觉错误信息可能会对国家安全、社会凝聚力和公共健康构成重大威胁。然而，我们需要更多地了解特定的视觉特征，如颜色和面部存在，可能会影响人们对此类视觉帖子的可信度的评估。该项目全面了解了不同的视觉元素如何影响用户对图像和视频的可信度。研究结果帮助平台和事实核查机构发现视觉错误信息，遏制其扩散，识别脆弱的用户群体，并制定纠正干预措施。本研究项目广泛借鉴计算机科学、广告、营销、认知科学和传播学的文献，使用计算机视觉分析、定性访谈、大规模人类注释和实验，旨在：1)识别可能影响人们可信度感知的特定视觉特征和机制；2)检查这些视觉特征如何与非视觉特征(来源、病毒等)和用户特征(党派意识、数字媒体素养等)相互作用，以及3)研究如何有效地利用这些视觉特征来纠正错误信息。研究小组正在汇编一个大规模的开放数据集，其中包括带有人类注释的视觉帖子。虽然现有的错误信息数据集主要关注信息的真实性，但该数据集提供了可信度感知以及其他相关结果，如注意力、情感反应和审美吸引力。此外，研究小组正在创建一个网站，提供可访问的信息，教育公众有关图像和视频中的错误信息，以便公众能够意识到他们的漏洞，并对某些类型的视觉信息更加警惕。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Today’s misinformation posts have increasingly been presented in visual formats, such as images, memes, and videos. Compared to text, visuals are processed faster, remembered better, and are more likely to be shared on social media. As technology makes image and video manipulation accessible to the masses, visual misinformation can be a significant threat to national security, social cohesion, and public health. Yet we need to know more about how specific visual features, such as color and face presence, may influence how people evaluate the credibility of such visual posts. This project offers a comprehensive understanding of how different visual elements may influence users’ perceived credibility of images and videos. The results help platforms and fact-checking agencies to detect visual misinformation, curb its diffusion, identify vulnerable user groups, and develop corrective interventions.Drawing broadly from literature in computer science, advertising, marketing, cognitive science, and communication, and using computer vision analysis, qualitative interviews, large-scale human annotation, and experiments, this research project aims to: 1) identify the specific visual features and mechanisms which may influence people’s credibility perceptions, 2) examine how these visual features interact with non-visual features (source, virality, etc) and user characteristics (partisanship, digital media literacy, etc), and 3) examine how these visual features can be effectively leveraged in misinformation correction efforts. The research team is compiling a large-scale open dataset of visual posts with human annotations. While existing misinformation datasets have largely focused on the veracity of messages, this dataset provides credibility perceptions along with other relevant outcomes such as attention, emotional reactions and aesthetic appeal. In addition, the research team is creating a website with accessible information to educate the general public about misinformation presented in images and videos, so that the public can be aware of their vulnerabilities and be more vigilant towards certain types of visual information.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.