Collaborative Research: SaTC: CORE: Small: Understanding how visual features of misinformation influence credibility perceptions

如今的错误信息越来越多地以视觉形式呈现，比如图片、表情包和视频。与文字相比，视觉效果处理得更快，记忆得更好，也更有可能在社交媒体上分享。随着技术使大众能够操纵图像和视频，视觉上的错误信息可能对国家安全、社会凝聚力和公共卫生构成重大威胁。然而，我们需要更多地了解具体的视觉特征，如颜色和面部表情，可能会影响人们如何评估这些视觉帖子的可信度。该项目提供了对不同视觉元素如何影响用户对图像和视频的感知可信度的全面理解。研究结果有助于平台和事实核查机构发现视觉上的错误信息，遏制其传播，识别弱势用户群体，并制定纠正措施。广泛参考计算机科学、广告、市场营销、认知科学和传播学方面的文献，并采用计算机视觉分析、定性访谈、大规模人类注释和实验，本研究项目旨在：1)确定可能影响人们可信度感知的特定视觉特征和机制，2)检查这些视觉特征如何与非视觉特征（来源，病毒性等）和用户特征（党派关系，数字媒体素养等）相互作用，以及3)检查如何有效利用这些视觉特征在错误信息纠正工作中。研究小组正在编制一个大规模的开放数据集，其中包含人工注释的视觉帖子。虽然现有的错误信息数据集主要集中在信息的准确性上，但该数据集提供了可信度感知以及其他相关结果，如注意力、情绪反应和审美吸引力。此外，研究小组正在创建一个网站，提供可访问的信息，以教育公众关于图像和视频中呈现的错误信息，以便公众可以意识到它们的脆弱性，并对某些类型的视觉信息更加警惕。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Today’s misinformation posts have increasingly been presented in visual formats, such as images, memes, and videos. Compared to text, visuals are processed faster, remembered better, and are more likely to be shared on social media. As technology makes image and video manipulation accessible to the masses, visual misinformation can be a significant threat to national security, social cohesion, and public health. Yet we need to know more about how specific visual features, such as color and face presence, may influence how people evaluate the credibility of such visual posts. This project offers a comprehensive understanding of how different visual elements may influence users’ perceived credibility of images and videos. The results help platforms and fact-checking agencies to detect visual misinformation, curb its diffusion, identify vulnerable user groups, and develop corrective interventions.Drawing broadly from literature in computer science, advertising, marketing, cognitive science, and communication, and using computer vision analysis, qualitative interviews, large-scale human annotation, and experiments, this research project aims to: 1) identify the specific visual features and mechanisms which may influence people’s credibility perceptions, 2) examine how these visual features interact with non-visual features (source, virality, etc) and user characteristics (partisanship, digital media literacy, etc), and 3) examine how these visual features can be effectively leveraged in misinformation correction efforts. The research team is compiling a large-scale open dataset of visual posts with human annotations. While existing misinformation datasets have largely focused on the veracity of messages, this dataset provides credibility perceptions along with other relevant outcomes such as attention, emotional reactions and aesthetic appeal. In addition, the research team is creating a website with accessible information to educate the general public about misinformation presented in images and videos, so that the public can be aware of their vulnerabilities and be more vigilant towards certain types of visual information.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.