Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data.
Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data.
复制标题
用于提高众包情感数据质量的概率多图建模。
DOI：
10.1109/taffc.2017.2678472
复制
发表时间：
2019
影响因子：
11.2
通讯作者：
Wang,JamesZ
中科院分区：
文献类型：
--
作者：
Ye,Jianbo;Li,Jia;Newman,MichelleG;AdamsJr,ReginaldB;Wang,JamesZ
我们提出了一种概率方法，在众包情感研究中联合建模参与者的可靠性和人类的规律性。可靠性衡量的是受试者认真回答问题的可能性；规律性衡量的是一个人对来自目标人群的其他认真输入的回应表示同意的频率。基于众包的研究或实验依赖于人类自我报告的情感，与试图获得物体的具体非情感标签的典型众包研究相比，提出了额外的挑战。在典型的非情感众包研究中，参与者的可靠性被大量追求，而人类在情感实验中的规律性本身并没有被彻底考虑。人们经常观察到，不同的人在同一个测试问题上表现出不同的感受，这并不是唯一正确的答案。因此，一个人的回答的高可靠性不能最终导致个人之间的高度共识。相反，研究人员感兴趣的是在全球范围内测试人群的共识。我们的概率模型建立在任务和工作者之间的一致多重图的基础上，区分了主体规律性和总体可靠性。我们证明了该方法对大规模众包情感数据的深入稳健分析的有效性，包括通过向人类受试者呈现视觉刺激而收集的情感和审美评估。
We proposed a probabilistic approach to joint modeling of participants' reliability and humans' regularity in crowdsourced affective studies. Reliability measures how likely a subject will respond to a question seriously; and regularity measures how often a human will agree with other seriously-entered responses coming from a targeted population. Crowdsourcing-based studies or experiments, which rely on human self-reported affect, pose additional challenges as compared with typical crowdsourcing studies that attempt to acquire concrete non-affective labels of objects. The reliability of participants has been massively pursued for typical non-affective crowdsourcing studies, whereas the regularity of humans in an affective experiment in its own right has not been thoroughly considered. It has been often observed that different individuals exhibit different feelings on the same test question, which does not have a sole correct response in the first place. High reliability of responses from one individual thus cannot conclusively result in high consensus across individuals. Instead, globally testing consensus of a population is of interest to investigators. Built upon the agreement multigraph among tasks and workers, our probabilistic model differentiates subject regularity from population reliability. We demonstrate the method's effectiveness for in-depth robust analysis of large-scale crowdsourced affective data, including emotion and aesthetic assessments collected by presenting visual stimuli to human subjects.