EAGER: DCL: SaTC: Enabling Interdisciplinary Collaboration: Improving Human Discernment of Audio Deepfakes via Multi-level Information Augmentation

EAGER：DCL：SaTC：实现跨学科合作：通过多级信息增强提高人类对音频深赝品的识别能力

• 批准号: 2210011
• 负责人: Vandana Janeja
• 金额: $ 29.98万
• 依托单位: University of Maryland Baltimore County
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-01 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2210011&HistoricalAwards=false
• 关键词: EAGER; DCL; SaTC; Enabling; Interdisciplinary

This project increases listeners’ discernment of audio deepfakes through augmentation of information, both technological and sociolinguistic. This project establishes an innovative pathway for collaborative research across sociolinguistics, human centered analytics, and data science and lays the groundwork for future analyses of deepfakes that are broadly relevant across disciplines, informed by human behavioral perspectives. The project will address the societal challenge of misinformation by generating insights that can increase the ability of listeners – particularly college students, whose lives are indelibly shaped by technology – to evaluate the veracity and authenticity of information online. The project's broader significance is to address the societal challenge of misinformation by generating insights that can help empower listeners to make decisions about how to evaluate the veracity and authenticity of information they encounter online. The project improves understanding and modeling of how deepfakes are involved in spreading misinformation and tracking how language technology is adapted for social harm and/or used in unethical ways. The proposed work will increase listeners’ discernment of audio deepfakes through augmentation of information that draws upon integrated interdisciplinary knowledge and advances data augmentation as an important tool for deepfake detection. The objectives of the project are to: (1) Study and evaluate listener perceptions of audio deepfakes that have been created with varying degrees of linguistic complexity; (2) Study and evaluate the efficacy of training sessions that increase listeners’ sociolinguistic perceptual ability and improve their ability to discern deepfake audio content; (3) Augment the audio deepfake discernment via multi-level temporal and linguistic signatures, informed by training and linguistic labeling; (4) Evaluate the impact of augmented signature information on listener perceptions of audio deepfakes; (5) Create open-access online modules and materials with social science and data science student involvement to improve listeners’ discernment of audio cues on a wider public scale.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目通过技术和社会语言学两方面的信息增强了听众对音频深度伪造的辨别能力。该项目为跨社会语言学、以人为中心的分析学和数据科学的合作研究建立了一条创新途径，并为未来基于人类行为视角的跨学科广泛相关的深度造假分析奠定了基础。该项目将通过产生见解来解决错误信息带来的社会挑战，这些见解可以提高听众——特别是那些生活不可磨灭地受到技术影响的大学生——评估在线信息的真实性和真实性的能力。该项目更广泛的意义在于，通过产生见解，帮助听众决定如何评估他们在网上遇到的信息的真实性，从而解决错误信息带来的社会挑战。该项目提高了对深度造假如何参与传播错误信息的理解和建模，并跟踪语言技术如何被用于社会危害和/或以不道德的方式使用。提出的工作将通过利用综合跨学科知识增强信息来提高听众对音频深度伪造的识别能力，并将数据增强作为深度伪造检测的重要工具。该项目的目标是：(1)研究和评估听众对不同语言复杂性程度的音频深度伪造的感知；(2)研究和评估提高听者社会语言学感知能力和识别深度虚假音频内容能力的培训课程的效果；(3)通过训练和语言标注，通过多层次的时间和语言签名增强音频深度识别；(4)评估增强签名信息对听者对音频深度伪造感知的影响；(5)创建开放获取的在线模块和材料，让社会科学和数据科学学生参与其中，以提高听众在更广泛的公众范围内对音频线索的识别能力。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Auto Annotation of Linguistic Features for Audio Deepfake Discernment

• DOI: 10.1609/aaaiss.v2i1.27682
• 发表时间: 2024-01
• 期刊: Proceedings of the AAAI Symposium Series
• 作者: Kifekachukwu Nwosu;Chloe Evered;Zahra Khanjani;Noshaba Bhalli;Lavon Davis;Christine Mallinson;V. P. Janeja

Learning to Listen and Listening to Learn: Spoofed Audio Detection Through Linguistic Data Augmentation

• DOI: 10.1109/isi58743.2023.10297267
• 发表时间: 2023-10
• 期刊: 2023 IEEE International Conference on Intelligence and Security Informatics (ISI)
• 作者: Zahra Khanjani;Lavon Davis;Anna Tuz;Kifekachukwu Nwosu;Christine Mallinson;V. P. Janeja