Audio-visual prosody of whispered and semi-whispered speech

韵律在言语中起着强大的作用。它帮助听众识别单词和短语的界限，它突出最重要的信息片段，并对情感进行编码。浊音韵律是语音学和音系学的一个重要研究课题，但迄今为止对浊音韵律的研究还很少。更重要的是，半耳语中的韵律还没有被研究。本课题的主要目的是对耳语和半耳语的韵律提供更多的洞察。这一目标的挑战性在于，作为韵律变异的主要相关因素之一的基频在耳语中完全不存在。因此，我们提出了一种通过其他声学参数来分析韵律的方法，例如元音和辅音的强度、持续时间和频谱特性。在我们的分析中，我们特别关注韵律和辅音之间的交互作用，并利用运动跟踪技术的可能性，将所研究的参数空间扩展到手势维度。计划中的实验包括对比仅在声学通道和在联合声光通道中进行交流的条件。我们感兴趣的是在这些情况下声学提示的差异程度，以及视觉韵律如何以及是否在(半)耳语中增强或取代听觉提示。最后，拟议的调查为一个更大的企业铺平了道路，其最终目标是设计将半耳语和耳语语音转换为有声语音的算法。在本项目中，我们专注于半耳语语音，目标是将声学研究的结果在一个半耳语到语音转换系统中实现，该系统将在非全语音(Ilse语料库)上进行测试。最终，这样的系统还将有助于缓解老龄化和患有喉癌等疾病或包括帕金森氏症在内的神经退行性疾病的临床人口的日常挑战。

Prosody plays a powerful role in speech. It helps listeners identify the boundaries of words and phrases, it highlights the most important pieces of information and it encodes emotions. While voiced speech prosody is an established research topic in phonetics and phonology, the prosody of whispered speech has received little attention so far. What is more, prosody in semi-whispered speech has not yet been investigated. The main goal of the present project is to provide more insight into the prosody of whispered and semi-whispered speech.The challenging nature of this goal lies in the fact that fundamental frequency, one of the main correlates of prosodic variation, is entirely absent in whispered speech. Thus, we propose an analysis of prosody through other acoustic parameters such as intensity, duration, and spectral properties of vowels and consonants. In our analysis we pay special attention to the still understudied interaction of prosody with consonants.In addition, we aim to extend the studied parameter space to the gestural dimension by taking advantage of the possibilities of motion tracking technology. The planned experiments include conditions that contrast communication in the acoustic-only with the joint acoustic-and-visual channel. We are interested in the extent to which acoustic cues may differ in these conditions, as well as in how and if visual prosody augments or supplants auditory cues in (semi-)whispered speech.Finally, the proposed investigation paves the way for a larger enterprise whose ultimate goal is to design algorithms which will convert semi-whispered and whispered speech to voiced speech. In the present project we concentrate on semi-whispered speech and aim to implement the results of the acoustic study in a semi-whisper-to-voiced conversion system, to be tested on speech that is not fully voiced (ILSE Corpus). Ultimately such a system will also help relieve the daily challenges of the ageing and clinical populations suffering from ailments such as throat cancers or neuro-degenerative diseases including Parkinson’s.