RI: Small: Low-Latency and High-Quality Simultaneous Translation

同声传译被广泛应用于许多场合，包括联合国等多边组织、国际峰会和会议以及法律诉讼。然而，同时使用两种语言进行感知和生产使得这项任务对人类来说非常具有挑战性和累人。专业同声传译人员的数量在世界范围内极其有限，他们必须以两到三人一组的方式工作，每个译员只能维持大约15-30分钟。因此，迫切需要开发同声传译技术，以减轻人工口译人员的负担，并使这项服务更容易获得和负担得起。然而，同声翻译对于机器来说也是出了名的困难，持续可靠地完成同声翻译被认为是人工智能的圣杯之一。为了解决这个问题，人们提出了各种各样的方法，但都有三个主要的局限性：(a)他们的翻译模型仍然是一个完整的句子翻译模型；(b)它们不能实现短延迟，如人类口译中常见的“3秒延迟”；(c)他们的系统很复杂，很难训练。因此，本项目旨在开发新的算法、技术和数据集，以实现最小延迟（低延迟）的高质量同步机器翻译。该项目开发的技术将使同声传译更加经济实惠，更容易获得，这将提高人类跨越语言障碍的沟通效率。该项目还通过招募机器翻译研究人员来支持代表性不足的少数民族（母语不是英语的人）的STEM教育。在本项目的基础上，本项目的主要思想是摒弃传统的整句翻译范式和经典的序列到序列的翻译框架，这些框架在翻译之前处理完整的输入句子，因此不适合同声翻译。相反，这个项目采用了“前缀到前缀”的框架，模仿人工译员，只处理几个输入的单词就开始翻译。虽然非常简单，但该框架实现了低延迟和高翻译质量。使用这个框架，这个项目的目标是：(1)开发一种算法来检测和修复动态的预期错误，并探索新的评估指标，可以用于修订的翻译；(2)制定动态灵活的翻译策略，平衡质量和延迟；(3)通过修改平行文本中的参考译文，去除不必要的重排，构建更好的同声翻译训练数据；(4)将前缀到前缀框架应用于增量文本到语音合成（TTS），从而完成端到端同步语音到语音的管道，提高其质量和延迟，并与人工同声传译进行比较。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Simultaneous language translation (interpretation) is widely used in many situations including multilateral organizations such as the United Nations, international summits and conferences, and legal proceedings. However, the concurrent perception and production in two languages makes this task extremely challenging and exhausting for humans. The number of professional simultaneous interpreters is extremely limited worldwide, and they have to work in groups of two or three where each interpreter can only sustain for about 15-30 minutes. Therefore, there is a critical need to develop simultaneous translation techniques to reduce the burden of human interpreters and make this service more accessible and affordable. However, simultaneous translation is also notoriously difficult for machines and accomplishing it consistently and reliably is considered one of the holy grails of Artificial Intelligence. Various methods have been proposed to solve this problem, but with three major limitations: (a) their translation model is still a full-sentence translation model; (b) they cannot achieve short latencies such as "3-seconds delay" common in human interpretation; and (c) their systems are complicated and difficult to train. Therefore, this project aims to develop new algorithms, techniques, and datasets for high-quality simultaneous machine translation with minimum delay (low latency). The technologies developed by this project will make simultaneous translation more affordable and accessible, which will improve the efficiency of human communication across linguistic barriers. This project also supports STEM education of underrepresented minorities (who do not speak English natively) by recruiting them in machine translation studies.Based on the principal investigator's successful prior work, the key idea in this project is to discard the conventional full-sentence translation paradigm and the classical sequence-to-sequence framework which processes the full input sentence before starting to translate and are thus ill-suited to simultaneous translation. Instead, this project adopts a "prefix-to-prefix" framework which starts translation after processing only a few input words, mimicking human interpreters. Though extremely simple, this framework achieves low latency and high translation quality. Using this framework, this project aims to (1) Develop an algorithm to detect and fix anticipation mistakes on the fly, and explore new evaluation metrics that can work for translations with revisions; (2) Develop dynamic and flexible translation strategies to balance quality and latency; (3) Construct better training data for simultaneous translation by revising the reference translations in a parallel text to remove unnecessary reorderings; (4) Apply the prefix-to-prefix framework to incremental text-to-speech synthesis (TTS), thus completing the end-to-end simultaneous speech-to-speech pipeline, improve its quality and latency, and compare with human simultaneous interpreters.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.