Crowdsourcing for Multiple-Choice Question Answering
Crowdsourcing for Multiple-Choice Question Answering
复制标题
DOI：
10.1609/aaai.v28i2.19016
复制
发表时间：
2014-07
期刊：
影响因子：
--
通讯作者：
B. Aydin;Y. Yilmaz;Yaliang Li;Qi Li;Jing Gao;M. Demirbas
中科院分区：
文献类型：
--
作者：
B. Aydin;Y. Yilmaz;Yaliang Li;Qi Li;Jing Gao;M. Demirbas
我们利用大众智慧进行多项选择题回答，并采用轻量级机器学习技术来提高这些问题的众包答案的聚合准确性。为了开发更有效的聚合方法并进行经验评估，我们开发并部署了一个众包系统，用于玩“谁想成为百万富翁？”智力竞赛节目。分析我们的数据（由超过20万个答案组成），我们发现，通过选择聚合中最多的答案，我们可以正确回答90%以上的问题，但是对于智力竞赛节目中较晚/较难的问题，这种技术的成功率下降到60%。为了提高这些后来的/更难的问题的成功率，我们研究了新的加权聚合方案，用于聚合从人群中获得的答案。通过使用针对参与者的可靠性（来自参与者的信心）进行优化的权重，我们表明，我们可以将较难问题的准确率提高15%，总体平均准确率达到95%。我们的研究结果为应用机器学习技术构建更准确的众包问答系统的好处提供了一个很好的案例。
We leverage crowd wisdom for multiple-choice question answering, and employ lightweight machine learning techniques to improve the aggregation accuracy of crowdsourced answers to these questions. In order to develop more effective aggregation methods and evaluate them empirically, we developed and deployed a crowdsourced system for playing the “Who wants to be a millionaire?” quiz show. Analyzing our data (which consist of more than 200,000 answers), we find that by just going with the most selected answer in the aggregation, we can answer over 90% of the questions correctly, but the success rate of this technique plunges to 60% for the later/harder questions in the quiz show. To improve the success rates of these later/harder questions, we investigate novel weighted aggregation schemes for aggregating the answers obtained from the crowd. By using weights optimized for reliability of participants (derived from the participants’ confidence), we show that we can pull up the accuracy rate for the harder questions by 15%, and to overall 95% average accuracy. Our results provide a good case for the benefits of applying machine learning techniques for building more accurate crowdsourced question answering systems.