Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study
Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study
复制标题
DOI：
10.1109/vlhcc.2011.6070377
复制
发表时间：
2011-11
期刊：
影响因子：
--
通讯作者：
Amber Shinsel;Todd Kulesza;Margaret M. Burnett;William Curran;Alex Groce;Simone Stumpf;Weng-Keen Wong
中科院分区：
文献类型：
--
作者：
Amber Shinsel;Todd Kulesza;Margaret M. Burnett;William Curran;Alex Groce;Simone Stumpf;Weng-Keen Wong
智能助手有时处理太重要的任务，以至于不能被隐式地信任。最终用户可以通过系统评估建立信任，但这种评估成本很高。本文研究了是否，何时以及如何将一小群最终用户带到智能助理的评估上，从成本/效益的角度来看是有用的。我们的研究结果表明，一小群测试人员提供的好处比工作量的明显减少要多得多，但是这些好处并没有随着小群规模的增加而线性增加-有一个收益递减点，成本效益比变得不那么有吸引力。
Intelligent assistants sometimes handle tasks too important to be trusted implicitly. End users can establish trust via systematic assessment, but such assessment is costly. This paper investigates whether, when, and how bringing a small crowd of end users to bear on the assessment of an intelligent assistant is useful from a cost/benefit perspective. Our results show that a mini-crowd of testers supplied many more benefits than the obvious decrease in workload, but these benefits did not scale linearly as mini-crowd size increased - there was a point of diminishing returns where the cost-benefit ratio became less attractive.