Conversational Speech Quality Evaluation in the Crowd

现代电信服务的成功在很大程度上取决于其用户所感知的质量。对于包括IP语音的语音服务，感知质量在主观实验中被量化，在主观实验中，参与者要么收听通过被测系统传输的记录的语音样本，要么进行对话以获得更真实的评估结果。通常，评估是在实验室环境中进行的，该环境允许控制混杂因素，例如背景噪声或参与者的听力特征。国际电信联盟(国际电联)就如何进行这种实验室实验提出了建议，以便提供可靠的结果。然而，就成本和时间而言，它们需要付出很大的努力。众包(CS)通过提供一个通过互联网进行评估的全球参与者池，为质量评估提供了新的可能性。众包质量评估的潜在好处是调查a)由于用户群体的多样性而导致的参与者影响因素，b)由于参与者的现实生活环境而导致的环境影响因素，以及c)降低成本和周转时间。ITU-T建议P.808提供了使用CS评估进行语音质量评估的指南。然而，到目前为止，该方法仅限于听力测试，并没有提供任何关于会话测试的建议。在本项目中，我们旨在通过系统地回答以下关键研究问题来克服这一局限性：如何建立基于众包的会话测试，以提供有效和可靠的结果？具体地说，如何建立一个实时交流平台来控制所有考试外部因素(源于测试参与者的互联网连接或硬件)？如何在在线测试中评估测试参与者的特征和测试环境？测试程序应该如何设计？众包会话测试和实验室会话测试之间有哪些不同？研究问题将通过分析和量化受试者、他们的设备和测试环境最重要的特征在基于CS的会话测试中与标准实验室实验相比的影响来回答。此外，还将规定有效和可靠的检查方法，以远程分析相关特征，如环境噪声或听力。这项分析将为进行基于CS的会话测试提供数据驱动的指导方针，这些指导方针将被建议用于新的ITU-T建议，并将向研究界开放。

The success of modern telecommunication services largely depends on the quality perceived by their users. For speech services including Voice-over-IP, perceived quality is quantified in subjective experiments in which participants either listen to recorded speech samples transmitted through the system under test, or engage in a conversation to achieve a more a realistic evaluation result. Typically, evaluations are conducted in a laboratory environment which allows to control confounding factors, such as background noise or participant hearing characteristics. The International Telecommunication Union (ITU) has issued recommendations on how to carry out such laboratory experiments so that they deliver reliable results. However they invoke high efforts in terms of costs and time. Crowdsourcing (CS) offers new possibilities for quality assessment by providing a global pool of participants carrying out evaluations over the Internet. The potential benefits of crowdsourced quality evaluations are the investigation a) of participant influence factors due to the diverse population of users, b) of environment influence factors due to the real-life environment of the participants, and c) reduced costs and turnaround times. ITU-T Recommendation P.808 provides guidelines on using CS evaluations for speech quality assessment. However, the method is so far limited to the listening-only situation and does not provide any recommendation on conversation tests.In this project, we aim at overcoming this limitation by systematically answering the following key research questions: How should crowdsourcing-based conversation tests be set up in order to provide valid and reliable results? Specifically, how can a real-time communication platform be set up to control all test-external factors (stemming from the Internet connection or hardware of the test participants)? How can the characteristics of the test participants and the test environment be assessed in online tests? How should the test procedure be designed and which differences are to be expected between crowdsourcing- and laboratory-based conversation test?The research questions will be answered by analyzing and quantifying the impact of the most important characteristics of the participants, their device, and the test environment in CS-based conversation tests, in comparison to standard laboratory experiments. Furthermore, valid and reliable inspection methods will be specified to remotely analyze relevant characteristics, such as environmental noise or hearing abilities. The analysis will lead to data-driven guidelines for conducting CS-based conversation tests which will be proposed for a new ITU-T Recommendation, and will be openly available to the research community.