Evaluating the quality of speech services using crowdsourcing

最近，互联网服务的体验质量(QOE)越来越受到学术界和工业界的关注。对于语音通信服务，存在用于语音质量评估的长期确立的建议。通常，QOE是通过实验室环境中的主观实验来评估的，这允许控制混杂因素，如背景噪声条件。为了收集可靠的结果，研究人员和标准化机构，如国际电信联盟(ITU)，认为受控环境对QOE测量至关重要。然而，实验室实验在成本和时间方面都需要付出很高的努力。Crowdource(CS)为QOE研究提供了新的可能性，并为将QOE实验转移到互联网提供了一个全球参与者池。众包QOE研究的潜在好处是调查a)由于用户群体的多样性而导致的现实参与者影响因素，b)由于参与者的现实生活环境而导致的环境影响因素，以及c)降低成本和周转时间。ITU-T记录P.808关于使用CS方法进行语音质量评估，强调了参与者的特征、测试环境和回放系统对结果的有效性和可靠性的影响，并详细说明了一些期望的特征。然而，它并没有提供如何远程测试这些特征的指导，也没有提供这些特征的影响的定量评估。在本项目中，我们将在实验室和CS进行的大量实验的基础上，系统地回答以下关键研究问题：如何建立基于众包的语音质量评估实验，以提供有效和可靠的结果？具体来说，如何在网络测试中评估测试参与者、测试环境和回放系统的特征？基于众包的语音质量评估与实验室语音质量评估有哪些不同？如何影响这些差异对工具语音质量预测模型的发展？本研究项目的目的是分析在基于CS的语音质量评估中听者、听力设备和测试环境这三个最重要的特征的影响，并将它们的影响与标准的实验室实验进行比较。此外，将规定有效和可靠的测试方法，以远程分析相关特性(例如环境噪声)。这项分析将导致提出一项更新ITU-T记录的建议。第808页。此外，我们将在本项目框架中收集的基于CS的数据集(语音材料和主观分数)上评估现有用于预测语音质量的工具模型的性能。数据集和由此产生的建议将向研究界公开提供。

The Quality of Experience (QoE) of Internet services has recently gained increased interest by academia and industry. For speech communication services, long-established recommendations exist for speech quality evaluation. Typically, QoE is evaluated by means of subjective experiments in a laboratory environment, which allows to control confounding factors, such as background noise conditions. To collect reliable results, researchers and standardization bodies like the International Telecommunication Union (ITU) consider the controlled setting to be essential for QoE measurements. However, the laboratory experiments invoke high efforts in terms of costs and time.Crowdsourcing (CS) offers new possibilities for QoE research and provides a global pool of participants for shifting QoE experiments into the Internet. The potential benefits of crowdsourced QoE studies are the investigation a) of realistic participant influence factors due to the diverse population of users, b) of environmental influence factors due to the real-life environment of the participants, and c) reduced costs and turnaround times. ITU-T Rec. P.808 on using a CS approach for speech quality assessment emphasizes the influence of the participants’ characteristics, the test environment and the playback system on the validity and reliability of results, and details some desired characteristics. However, it does not provide guidance on how to remotely test those characteristics, nor it provides a quantitative assessment of the impact of those characteristics.In this project, we will systematically answer the following key research questions based on numerous experiments conducted in laboratory and CS: How should crowdsourcing-based speech quality evaluation experiments be set up to provide valid and reliable results? Specifically, how can the characteristics of the test participants, the test environment and the playback system be assessed in online tests? Which differences are to be expected between crowdsourcing-based and laboratory speech quality evaluation? In which way affect these differences the development of instrumental speech quality prediction models?It is the aim of the present research project to analyze the impact of the most important characteristics of the listener, the listening device, and the test environment in CS-based speech quality assessment, and to quantify their impact in comparison to standard laboratory experiments. Furthermore, valid and reliable test methods will be specified to remotely analyze relevant characteristics (e.g. environmental noise). The analysis will lead to a proposal for updating ITU-T Rec. P.808. In addition, we will evaluate the performance of existing instrumental models for predicting speech quality on the CS-based datasets (speech materials and subjective scores) collected in the frame of this project. The datasets and resulting recommendations will be made openly available to the research community.