CRII: SaTC: RUI: A Cross-Verification Approach for Identifying Tampered Audio

最近的技术进步使得制作名人做或说他们没有做或说的事情的假视频成为可能。有人担心，这项技术可能会被用来通过诽谤候选人来影响选举，或者通过公职人员的虚假陈述来煽动内乱。这个项目的重点是保护世界领导人免受这种虚假模仿。它从以历史为中心的角度来处理这个问题，提出了这样一个问题：“这个视频是一个可以从历史上证实的事件吗？”正如历史学家通过与其他主要来源的交叉核对来检验历史主张一样，我们也可以通过与其他主要视听信息来源的交叉核对来检验视听记录的真实性。该项目开发了这样一种交叉验证方法，用于识别虚假或篡改的音频内容。该奖项将支持一所高度多样化的文理学院的本科生研究。这个项目调查了两种情况下的音频交叉验证。在第一个场景中，目标是交叉验证具有可信来源记录的查询，该记录被认为是可靠的。为了识别通过插入、删除或修改进行的篡改，可以使用动态时间规整(DTW)来计算查询和源记录之间的对准。在这一方案中的关键贡献是发展了对如何在培训期间使用适当的平稳放松DTW将DTW纳入端到端学习系统的理解。在第二种情况下，目标是交叉验证具有不可信来源记录的查询，这些记录本身可能被篡改。在第二个场景中的关键贡献是扩展现有的对齐技术，以在存在恶意篡改的情况下联合对齐录音集合，并重建音频地面真实。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Recent advances in technology have made it possible to create fake videos of celebrities doing or saying things they did not do or say. There is concern that this technology may be used to influence elections by defaming candidates or to instigate civil unrest through false statements by public officials. This project focuses on protecting world leaders from such fake impersonations. It approaches the problem from a history-centric perspective by asking the question, “Is this video a historically verifiable event?” In the same way that a historian tests a historical claim by cross-checking against other primary sources, we can test the authenticity of an audiovisual recording by cross-checking against other primary sources of audiovisual information. This project develops such a cross-verification approach for identifying fake or tampered audio content. The award will support undergraduate research at a highly diverse liberal arts college.This project investigates audio cross-verification in two scenarios. In the first scenario, the goal is to cross-verify a query with a trusted source recording, which is assumed to be reliable. In order to identify tampering via insertion, deletion, or modification, the alignment between the query and source recording can be computed using dynamic time warping (DTW). The key contribution in this scenario is developing an understanding of how to incorporate DTW into an end-to-end learning system using an appropriate smooth relaxation of DTW during training. In the second scenario, the goal is to cross-verify a query with untrusted source recordings, which may themselves be tampered. The key contribution in the second scenario is to extend existing alignment techniques to jointly align a collection of recordings in the presence of malicious tampering, and to reconstruct the audio ground truth.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.