Crowdsourcing a normative natural language dataset: a comparison of Amazon Mechanical Turk and in-lab data collection.
Crowdsourcing a normative natural language dataset: a comparison of Amazon Mechanical Turk and in-lab data collection.
复制标题
DOI：
10.2196/jmir.2620
复制
发表时间：
2013-05-20
影响因子：
7.4
通讯作者：
Woods RL
中科院分区：
文献类型：
--
作者：
Saunders DR;Bex PJ;Woods RL
众包已经成为收集医学研究数据的一种有价值的方法。这种通过在网上公开征集人才的方法，对于收集大型标准数据集特别有用。然而，目前尚不清楚通过网络收集的自然语言数据集与在受控实验室条件下收集的自然语言数据集有何不同。将从众包参与者样本中获得的自然语言反应与在常规实验室环境中收集的根据特定年龄和性别标准招募的参与者的反应进行比较。我们收集了200个半分钟的电影剪辑的自然语言描述，来自Amazon机械土耳其工人(众包)和从社区招募的60名参与者(实验室来源)。众包参与者对他们想要的片段做出反应，并输入他们的答案，而实验室来源的参与者对40个片段进行口头回应，他们的回应被转录下来。这些回答的内容是使用一次取出来的程序进行评估的，该程序将对同一剪辑和其他剪辑的其他答复的答复与共享单词的平均数量进行比较。与从60个实验室来源的参与者(具有特定的人口统计特征)收集标准数据需要13个月的招聘相比，只需要34天就可以从99个众包参与者那里收集标准数据(提供22份答复的中位数)。大多数众包工作者是女性，年龄中位数为35岁，低于实验室来源的中位数62岁，但与美国人口的中位年龄相似。众包参与者的回答平均更长，即33个单词，而不是28个单词(P<；.001)，而且他们使用的词汇种类较少。然而，正如共享单词的跨数据集计数(P<；.001)所显示的那样，用于描述两个数据集之间的特定片段的单词有很强的相似性。在这两个数据集中，回复包含大量相关内容，与对同一片段的回复的共同词多于对其他片段的回复(P<；.001)。有证据表明，女性和年龄较大的众包参与者的回答中有更多的共享单词(P=.004和.01)，而年轻参与者在实验室来源的人群中有更多的共享单词(P=.01)。众包是一种快速、经济地收集大量可靠的标准自然语言回答数据集的有效方法。
Crowdsourcing has become a valuable method for collecting medical research data. This approach, recruiting through open calls on the Web, is particularly useful for assembling large normative datasets. However, it is not known how natural language datasets collected over the Web differ from those collected under controlled laboratory conditions. To compare the natural language responses obtained from a crowdsourced sample of participants with responses collected in a conventional laboratory setting from participants recruited according to specific age and gender criteria. We collected natural language descriptions of 200 half-minute movie clips, from Amazon Mechanical Turk workers (crowdsourced) and 60 participants recruited from the community (lab-sourced). Crowdsourced participants responded to as many clips as they wanted and typed their responses, whereas lab-sourced participants gave spoken responses to 40 clips, and their responses were transcribed. The content of the responses was evaluated using a take-one-out procedure, which compared responses to other responses to the same clip and to other clips, with a comparison of the average number of shared words. In contrast to the 13 months of recruiting that was required to collect normative data from 60 lab-sourced participants (with specific demographic characteristics), only 34 days were needed to collect normative data from 99 crowdsourced participants (contributing a median of 22 responses). The majority of crowdsourced workers were female, and the median age was 35 years, lower than the lab-sourced median of 62 years but similar to the median age of the US population. The responses contributed by the crowdsourced participants were longer on average, that is, 33 words compared to 28 words (P<.001), and they used a less varied vocabulary. However, there was strong similarity in the words used to describe a particular clip between the two datasets, as a cross-dataset count of shared words showed (P<.001). Within both datasets, responses contained substantial relevant content, with more words in common with responses to the same clip than to other clips (P<.001). There was evidence that responses from female and older crowdsourced participants had more shared words (P=.004 and .01 respectively), whereas younger participants had higher numbers of shared words in the lab-sourced population (P=.01). Crowdsourcing is an effective approach to quickly and economically collect a large reliable dataset of normative natural language responses.
登录
查看更多内容
影响因子：
7.4
作者：
Frost J;Okun S;Vaughan T;Heywood J;Wicks P
通讯作者：
Wicks P
影响因子：
3.7
作者：
Tung JY;Do CB;Hinds DA;Kiefer AK;Macpherson JM;Chowdry AB;Francke U;Naughton BT;Mountain JL;Wojcicki A;Eriksson N
通讯作者：
Eriksson N
影响因子：
16.4
作者：
Kraut, R;Olson, J;Couper, M
通讯作者：
Couper, M
影响因子：
5.4
作者：
Janssen, Dirk P.
通讯作者：
Janssen, Dirk P.
影响因子：
7.4
作者：
Turner, Anne M.;Kirchhoff, Katrin;Capurro, Daniel
通讯作者：
Capurro, Daniel