Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization
Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization
复制标题
DOI：
10.48550/arxiv.2305.11095
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Puyuan Peng;Brian Yan;Shinji Watanabe;David F. Harwath
中科院分区：
文献类型：
--
作者：
Puyuan Peng;Brian Yan;Shinji Watanabe;David F. Harwath
我们研究了最近提出的网络规模语音模型 Whisper 的新兴能力，通过快速工程使其适应看不见的任务。我们选择了三个任务：视听语音识别（AVSR）、代码转换语音识别（CS-ASR）和未见过的语言对上的语音翻译（ST）。我们通过利用另一个大型模型或简单地操作默认提示中的特殊标记来设计特定于任务的提示。实验表明，与默认提示相比，我们提出的提示在三个零样本任务上的性能提高了 10% 到 45%，甚至在某些数据集上优于 SotA 监督模型。此外，我们的实验揭示了 Whisper 的许多有趣的特性，包括它对提示的鲁棒性、对口音的偏见以及其潜在空间的多语言理解。代码可在 https://github.com/jasonppy/PromptingWhisper 获取
We investigate the emergent abilities of the recently proposed web-scale speech model Whisper, by adapting it to unseen tasks with prompt engineering. We selected three tasks: audio-visual speech recognition (AVSR), code-switched speech recognition (CS-ASR), and speech translation (ST) on unseen language pairs. We design task-specific prompts, by either leveraging another large-scale model, or simply manipulating the special tokens in the default prompts. Experiments show that compared to the default prompts, our proposed prompts improve performance by 10% to 45% on the three zero-shot tasks, and even outperform SotA supervised models on some datasets. In addition, our experiments reveal many interesting properties of Whisper, including its robustness to prompts, bias on accents, and the multilingual understanding in its latent space. Code is available at https://github.com/jasonppy/PromptingWhisper