RI: Small: Low-Latency and High-Quality Simultaneous Translation

同声传译广泛应用于联合国等多边组织、国际首脑会议、法律的诉讼等场合。然而，两种语言的同时感知和生产使得这项任务对人类来说极具挑战性和疲惫不堪。全世界专业同声传译人员的数量非常有限，他们必须以两到三人为一组工作，每个口译员只能维持大约15-30分钟。因此，迫切需要开发同声传译技术，以减轻口译员的负担，并使这项服务更容易获得和负担得起。然而，同声翻译对于机器来说也是出了名的困难，一致可靠地完成它被认为是人工智能的圣杯之一。已经提出了各种方法来解决这个问题，但有三个主要的限制：（a）他们的翻译模型仍然是一个完整的句子翻译模型;（B）他们不能实现短的延迟，如“3秒延迟”在人类翻译中常见的;和（c）他们的系统是复杂的，难以训练。 因此，该项目旨在开发新的算法，技术和数据集，以最小延迟（低延迟）实现高质量的同步机器翻译。该项目开发的技术将使同声传译更经济实惠，更容易获得，这将提高跨越语言障碍的人类沟通效率。该项目还支持代表性不足的少数民族的STEM教育（母语不是英语的人）通过招募他们进行机器翻译研究。基于主要研究者先前的成功工作，该项目的核心思想是摒弃传统的整句翻译范式和经典的顺序到顺序框架，该框架在开始翻译之前处理完整的输入句子，因此是病态的。适合同声传译。相反，这个项目采用了一个“前缀到前缀”的框架，在处理了几个输入单词后就开始翻译，模仿人类翻译。虽然非常简单，但该框架实现了低延迟和高翻译质量。使用这个框架，这个项目的目标是（1）开发一个算法来检测和修复预测错误的飞行，并探索新的评估指标，可以工作的翻译修订;（2）开发动态和灵活的翻译策略，以平衡质量和延迟;（三）通过修改平行文本中的参考译文来删除不必要的翻译，重新排序;（4）将前缀到前缀框架应用于增量式文语合成（TTS），从而完成端到端的同步语音到语音流水线，改善其质量和延迟，该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查进行评估，被认为值得支持的搜索.

Simultaneous language translation (interpretation) is widely used in many situations including multilateral organizations such as the United Nations, international summits and conferences, and legal proceedings. However, the concurrent perception and production in two languages makes this task extremely challenging and exhausting for humans. The number of professional simultaneous interpreters is extremely limited worldwide, and they have to work in groups of two or three where each interpreter can only sustain for about 15-30 minutes. Therefore, there is a critical need to develop simultaneous translation techniques to reduce the burden of human interpreters and make this service more accessible and affordable. However, simultaneous translation is also notoriously difficult for machines and accomplishing it consistently and reliably is considered one of the holy grails of Artificial Intelligence. Various methods have been proposed to solve this problem, but with three major limitations: (a) their translation model is still a full-sentence translation model; (b) they cannot achieve short latencies such as "3-seconds delay" common in human interpretation; and (c) their systems are complicated and difficult to train. Therefore, this project aims to develop new algorithms, techniques, and datasets for high-quality simultaneous machine translation with minimum delay (low latency). The technologies developed by this project will make simultaneous translation more affordable and accessible, which will improve the efficiency of human communication across linguistic barriers. This project also supports STEM education of underrepresented minorities (who do not speak English natively) by recruiting them in machine translation studies.Based on the principal investigator's successful prior work, the key idea in this project is to discard the conventional full-sentence translation paradigm and the classical sequence-to-sequence framework which processes the full input sentence before starting to translate and are thus ill-suited to simultaneous translation. Instead, this project adopts a "prefix-to-prefix" framework which starts translation after processing only a few input words, mimicking human interpreters. Though extremely simple, this framework achieves low latency and high translation quality. Using this framework, this project aims to (1) Develop an algorithm to detect and fix anticipation mistakes on the fly, and explore new evaluation metrics that can work for translations with revisions; (2) Develop dynamic and flexible translation strategies to balance quality and latency; (3) Construct better training data for simultaneous translation by revising the reference translations in a parallel text to remove unnecessary reorderings; (4) Apply the prefix-to-prefix framework to incremental text-to-speech synthesis (TTS), thus completing the end-to-end simultaneous speech-to-speech pipeline, improve its quality and latency, and compare with human simultaneous interpreters.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.