CRII: SaTC: RUI: A Cross-Verification Approach for Identifying Tampered Audio

最近的技术进步使得制作名人做或说他们没有做或说的事情的假视频成为可能。有人担心，这种技术可能被用来通过诽谤候选人来影响选举，或通过公职人员的虚假陈述来煽动内乱。这个项目的重点是保护世界各国领导人免受此类假冒行为的影响。它从以历史为中心的角度来解决这个问题，提出这样一个问题：“这个视频是一个历史上可证实的事件吗？”就像历史学家通过交叉核对其他原始资料来检验一个历史论断一样，我们也可以通过交叉核对其他原始音像资料来检验一段音像记录的真实性。本项目开发了这样一种交叉验证方法来识别虚假或篡改的音频内容。该奖项将支持一所高度多元化的文理学院的本科生研究。本项目在两种情况下研究音频交叉验证。在第一个场景中，目标是用可信源记录交叉验证查询，假设可信源记录是可靠的。为了识别通过插入、删除或修改进行的篡改，可以使用动态时间规整（DTW）计算查询和源记录之间的对齐。在这个场景中，关键的贡献是理解如何在训练期间使用适当的平滑放松DTW将DTW纳入端到端学习系统。在第二个场景中，目标是用不可信的源记录交叉验证查询，这些记录本身可能被篡改。第二种方案的关键贡献是扩展现有的对齐技术，以便在存在恶意篡改的情况下联合对齐一组录音，并重建音频的地面真实性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Recent advances in technology have made it possible to create fake videos of celebrities doing or saying things they did not do or say. There is concern that this technology may be used to influence elections by defaming candidates or to instigate civil unrest through false statements by public officials. This project focuses on protecting world leaders from such fake impersonations. It approaches the problem from a history-centric perspective by asking the question, “Is this video a historically verifiable event?” In the same way that a historian tests a historical claim by cross-checking against other primary sources, we can test the authenticity of an audiovisual recording by cross-checking against other primary sources of audiovisual information. This project develops such a cross-verification approach for identifying fake or tampered audio content. The award will support undergraduate research at a highly diverse liberal arts college.This project investigates audio cross-verification in two scenarios. In the first scenario, the goal is to cross-verify a query with a trusted source recording, which is assumed to be reliable. In order to identify tampering via insertion, deletion, or modification, the alignment between the query and source recording can be computed using dynamic time warping (DTW). The key contribution in this scenario is developing an understanding of how to incorporate DTW into an end-to-end learning system using an appropriate smooth relaxation of DTW during training. In the second scenario, the goal is to cross-verify a query with untrusted source recordings, which may themselves be tampered. The key contribution in the second scenario is to extend existing alignment techniques to jointly align a collection of recordings in the presence of malicious tampering, and to reconstruct the audio ground truth.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.