Experiments and models of speech recognition across tonal and non-tonal language systems (EMSATON)

人类的言语交际是我们文化的基础。尽管所有人的发音器官和耳朵都非常相似，但它们在不同语言中的使用在解决任务方面表现出高度的差异性，不仅在安静的情况下有效地进行沟通，而且在具有挑战性的声学条件下和听力障碍。目前的项目将揭示如何通过探索音调和非音调语言的语音识别的声学，语音学和听力学基础以及当前语音识别模型复制音调和非音调语言识别中可能存在的差异的能力来实现这一点。 EMSATON的主要长期目标是定量地了解人类语音识别在噪声中的减少，因为受到不同说话者和说话风格的影响（即，伦巴第语），不同的语言系统（即，声调语言（普通话，广东话）与西方语言（德语，英语，西班牙语））和不同的损害因素（即，我们将利用和扩展闭集多语言矩阵句子识别测试，该测试可用于以跨语言的高度可比的方式评估语音识别（即，20种语言，包括德语，英国和美国英语，西班牙语，最近还有普通话）。我们会发展广东话的矩阵测试，以第二种声调语言作为指涉，并会把新的声调语言测试与非声调语言挂钩。我们还将调查的影响，包括（双语）的谈话者和说话风格（正常和伦巴第语的讲话与高生产的努力）的影响。 将进行客观的声学语音分析和语音识别建模，以更好地了解不同语言（音调与非音调）、说话者和说话风格之间的差异和不同语音提示的重要性。 为了确定与下列因素有关的因素：为了评估不同语言之间的（语音相关）差异，并评估现有模型（如SII，HASPI，STOI或FADE模型）的一些假设，跨语言，说话者和说话风格的数据将用于测试当前模型的预测准确性，并建立一组基准数据和预测。这将为我们提供定量，基于模型的语言效果和几个潜在因素的分析，在两个声调语言（普通话，广东话）和典型的非声调语言（德语，英语，西班牙语）。一个可能的结果可能是以更具体的语言类型的方式构建辅助听力和听力设备的指导方针，从而优化音调和非音调语言使用者的各自利益。

Human speech communication is the basis of our culture. Even though the articulation organs and the ear are very similar across all humans, their usage across languages shows high variability in solving the task to effectively communicate not only in quiet, but also under challenging acoustical conditions and for hearing impairment. The current project will shed light on how this is achieved by exploring the acoustic, phonetic and audiological foundations of speech recognition of tonal and non-tonal languages and the ability of current speech recognition models to replicate possible differences in recognition across tonal and non-tonal languages. The main long-term goal of EMSATON is to quantitatively understand the reduction of human speech recognition in noise as being influenced by different talkers and speaking styles (i.e., Lombard speech), different language systems (i.e., tonal languages (Mandarin, Cantonese) vs. Western languages (German, English, Spanish)) and different impairment factors (i.e., type of noise, reverberation, individual hearing impairment).We will exploit and extend the closed-set multilingual Matrix sentence recognition test that can be used to assess speech recognition in a highly comparable way across languages (i.e., 20 languages including German, British and American English, Spanish, and recently Mandarin). We will develop the Matrix test in Cantonese to have a second tonal language as reference and will relate the new tonal language tests to non-tonal languages. We will also investigate the effect of talker by including (bilingual) talkers and the effect of speaking style (normal and Lombard speech with a high production effort). Both objective acoustic-phonetic analysis and speech recognition modelling will be performed to better understand the differences and the importance of different speech cues across different languages (tonal vs. non-tonal), across talkers, and speaking styles. In order to identify relevant factors for (speech-related) differences across very different languages and to evaluate a number of assumptions of existing models like SII, HASPI, STOI or the FADE model, the data across languages, speakers and speaking styles will be used to test the prediction accuracy of current models and to establish a benchmark set of data and predictions.This will provide us with the basis for a quantitative, model-based analysis of the language effect and several underlying factors across two tonal languages (Mandarin, Cantonese) and typical non-tonal languages (German, English, Spanish). A possible outcome might be guidelines for constructing assistive listening and hearing devices in a more language-type-specific way, thus optimizing the respective benefit for tonal and non-tonal language users.