Conversational Speech Quality Evaluation in the Crowd

现代电信服务的成功在很大程度上取决于其用户所感受到的质量。对于包括IP语音在内的语音服务，感知质量在主观实验中被量化，在主观实验中，参与者要么收听通过被测系统传输的记录语音样本，要么参与对话以获得更真实的评估结果。通常，评估在实验室环境中进行，该环境允许控制混杂因素，例如背景噪声或参与者听力特征。国际电信联盟（国际电联）已就如何进行此类实验室实验以提供可靠结果提出了建议。然而，它们在成本和时间方面需要付出很大的努力。众包为质量评估提供了新的可能性，它提供了一个通过互联网进行评价的全球参与者库。众包质量评价的潜在好处是调查a）由于用户群体的多样性而产生的参与者影响因素，B）由于参与者的现实生活环境而产生的环境影响因素，以及c）降低成本和周转时间。ITU-T建议书P.808提供了使用CS评估进行语音质量评估的指南。然而，到目前为止，该方法仅限于会话测试的情况下，并没有提供任何建议。在这个项目中，我们的目标是克服这一局限性，系统地回答以下关键研究问题：应该如何建立基于众包的会话测试，以提供有效和可靠的结果？具体来说，如何建立一个实时通信平台来控制所有测试外部因素（来自互联网连接或测试参与者的硬件）？如何在在线测试中评估测试参与者和测试环境的特征？测试程序应该如何设计，众包和实验室对话测试之间有哪些区别？研究问题将通过分析和量化参与者，他们的设备和测试环境中最重要的特征的影响来回答，在基于CS的会话测试中，与标准的实验室实验相比。此外，将规定有效和可靠的检查方法，以远程分析相关特征，如环境噪音或听力。该分析将产生数据驱动的指导方针，用于进行基于CS的会话测试，该指导方针将被提议用于新的ITU-T建议书，并将公开提供给研究界。

The success of modern telecommunication services largely depends on the quality perceived by their users. For speech services including Voice-over-IP, perceived quality is quantified in subjective experiments in which participants either listen to recorded speech samples transmitted through the system under test, or engage in a conversation to achieve a more a realistic evaluation result. Typically, evaluations are conducted in a laboratory environment which allows to control confounding factors, such as background noise or participant hearing characteristics. The International Telecommunication Union (ITU) has issued recommendations on how to carry out such laboratory experiments so that they deliver reliable results. However they invoke high efforts in terms of costs and time. Crowdsourcing (CS) offers new possibilities for quality assessment by providing a global pool of participants carrying out evaluations over the Internet. The potential benefits of crowdsourced quality evaluations are the investigation a) of participant influence factors due to the diverse population of users, b) of environment influence factors due to the real-life environment of the participants, and c) reduced costs and turnaround times. ITU-T Recommendation P.808 provides guidelines on using CS evaluations for speech quality assessment. However, the method is so far limited to the listening-only situation and does not provide any recommendation on conversation tests.In this project, we aim at overcoming this limitation by systematically answering the following key research questions: How should crowdsourcing-based conversation tests be set up in order to provide valid and reliable results? Specifically, how can a real-time communication platform be set up to control all test-external factors (stemming from the Internet connection or hardware of the test participants)? How can the characteristics of the test participants and the test environment be assessed in online tests? How should the test procedure be designed and which differences are to be expected between crowdsourcing- and laboratory-based conversation test?The research questions will be answered by analyzing and quantifying the impact of the most important characteristics of the participants, their device, and the test environment in CS-based conversation tests, in comparison to standard laboratory experiments. Furthermore, valid and reliable inspection methods will be specified to remotely analyze relevant characteristics, such as environmental noise or hearing abilities. The analysis will lead to data-driven guidelines for conducting CS-based conversation tests which will be proposed for a new ITU-T Recommendation, and will be openly available to the research community.