面向众包标注的真值推断与监督分类关键问题研究

近年来，数据标注任务可以方便地通过众包平台由互联网上的普通用户完成。由于众包标注者的标注质量参差不齐，利用众包标注数据进行监督学习模型训练充满挑战。本项目致力于研究众包标注数据机器学习中真值推断与监督分类问题，为基于众包标注数据构建监督分类学习系统提供新理论与新方法。首先，从经典的统计查询可学习理论出发，结合众包标注相关特性，研究预算约束条件下，样本特征、分类器、标注者质量与真值推断性能及学习模型质量之间的关系，建立指导真值推断和模型训练的基础理论；其次，研究基于样本“概念层”与“物理层”特征融合的多分类真值推断算法和“集成误标”样本标签纠正方法，以提高集成标签质量；再次，研究主动学习范式下更加精细的样本选择策略、标签优化方法以及基于标注质量时序建模的标注者选择策略，以提高学习模型的性能。最后，构建面向众包标注的真值推断与监督分类的原型系统，以验证研究成果的应用价值。

Recently, data annotation tasks can be completed by ordinary Internet users through crowdsourcing platforms. Since labeling qualities of labelers in crowdsourcing are different, supervised learning from these crowdsourced labeled data is full of challenges. This proposal focuses on ground truth inference and supervised learning for crowdsourcing from the aspect of machine learning, providing novel theories and methods for building supervised classification learning systems based on crowdsourced labeled data. Based on the classic statistical-query learnable theory and considering the specific characteristics of crowdsourcing, we first study the relationship among sample features, classifiers, labeling qualities, the performance of ground truth inference algorithms, and the qualities of learned models, and then build up a set of fundamental theories which can supervise ground truth inference and model training. Secondly, in order to improve the quality of integrated labels, we study novel algorithms for multi-class ground truth inference which are based on the fusion of concept-level features and physical-level features of examples, and novel methods which can correct integrated mislabeled examples. Thirdly, in order to improve the performance of learned models under an active learning paradigm, we study finer-grain sampling strategies, labeling optimization methods and labeler selection strategies based on temporal modeling for labeling qualities. Finally, we build up a prototype system for ground truth inference and supervised classification, using crowdsourced data to verify practical values of our research outcomes.