Adapting an ASR Foundation Model for Spoken Language Assessment
Adapting an ASR Foundation Model for Spoken Language Assessment
复制标题
DOI：
10.21437/slate.2023-20
复制
发表时间：
2023-07
期刊：
影响因子：
--
通讯作者：
Rao Ma;Mengjie Qian;M. Gales;K. Knill
中科院分区：
文献类型：
--
作者：
Rao Ma;Mengjie Qian;M. Gales;K. Knill
准确可靠的口语评估系统的关键部分是底层的ASR模型。最近，已经提供了大规模的预训练ASR基础模型，如Whisper。由于这些模型的输出被设计为人类可读，因此添加了标点符号，数字以阿拉伯数字形式呈现，并包括缩写。此外，这些模型往往会跳过输出中的不流畅和犹豫。虽然这些属性对可读性很有用，但对评估候选人的能力和提供反馈没有帮助。在这里，需要对候选人所说的话进行精确的转录。本文对Whisper输出进行了详细的分析，并提出了两种解决方案：微调和软提示调整。实验在公共语音语料库和英语学习者数据集上进行。结果表明，我们可以有效地改变耳语的解码行为，以产生准确的话说的反应。
A crucial part of an accurate and reliable spoken language assessment system is the underlying ASR model. Recently, large-scale pre-trained ASR foundation models such as Whisper have been made available. As the output of these models is designed to be human readable, punctuation is added, numbers are presented in Arabic numeric form and abbreviations are included. Additionally, these models have a tendency to skip disfluencies and hesitations in the output. Though useful for readability, these attributes are not helpful for assessing the ability of a candidate and providing feedback. Here a precise transcription of what a candidate said is needed. In this paper, we give a detailed analysis of Whisper outputs and propose two solutions: fine-tuning and soft prompt tuning. Experiments are conducted on both public speech corpora and an English learner dataset. Results show that we can effectively alter the decoding behaviour of Whisper to generate the exact words spoken in the response.