Automating creativity assessment with SemDis: An open platform for computing semantic distance.
Automating creativity assessment with SemDis: An open platform for computing semantic distance.
复制标题
DOI：
10.3758/s13428-020-01453-w
复制
发表时间：
2021-04
影响因子：
5.4
通讯作者：
Johnson DR
中科院分区：
文献类型：
--
作者：
Beaty RE;Johnson DR
创造力研究需要评估创意和产品的质量。在实践中，进行创造力研究通常需要几个人类评分员来判断参与者对创造力任务的反应，例如从交替使用任务（AUT）中判断想法的新奇。虽然这样的主观评分方法已被证明是有用的，他们有两个固有的局限性-劳动力成本（评分员通常编码成千上万的反应）和主观性（评分员不同的看法和偏好）-提高经典的心理测量的可靠性和有效性的威胁。我们试图解决主观评分的局限性，利用最近的发展，通过语义距离，一种计算方法，使用自然语言处理来量化文本的语义相关性的口头创造力的自动评分。在五项研究中，我们比较了表现最好的语义模型（例如，GloVe，连续的词袋），以前被证明具有最高的对应人类的相关性判断。我们评估了这些语义模型与人类的创造力评级从一个典型的口头创造力任务（AUT;研究1-3）和新奇/创造力评级从两个单词联想任务（研究4-5）。我们发现，一个潜在的语义距离因素，包括从五个语义模型的共同差异，可靠和强烈预测人类的创造力和新奇评级在一系列的创造性任务。我们还复制了创造力文献中的一个既定实验效应（即，序列顺序效应），并表明语义距离与其他创造性措施，证明收敛效度。我们提供了一个开放的平台来有效地计算语义距离，包括教程和文档（https：//osf.io/gz4fc/）。本文的在线版本（10.3758/s13428-020-01453-w）包含补充材料，可供授权用户使用。
Creativity research requires assessing the quality of ideas and products. In practice, conducting creativity research often involves asking several human raters to judge participants’ responses to creativity tasks, such as judging the novelty of ideas from the alternate uses task (AUT). Although such subjective scoring methods have proved useful, they have two inherent limitations—labor cost (raters typically code thousands of responses) and subjectivity (raters vary on their perceptions and preferences)—raising classic psychometric threats to reliability and validity. We sought to address the limitations of subjective scoring by capitalizing on recent developments in automated scoring of verbal creativity via semantic distance, a computational method that uses natural language processing to quantify the semantic relatedness of texts. In five studies, we compare the top performing semantic models (e.g., GloVe, continuous bag of words) previously shown to have the highest correspondence to human relatedness judgements. We assessed these semantic models in relation to human creativity ratings from a canonical verbal creativity task (AUT; Studies 1–3) and novelty/creativity ratings from two word association tasks (Studies 4–5). We find that a latent semantic distance factor—comprised of the common variance from five semantic models—reliably and strongly predicts human creativity and novelty ratings across a range of creativity tasks. We also replicate an established experimental effect in the creativity literature (i.e., the serial order effect) and show that semantic distance correlates with other creativity measures, demonstrating convergent validity. We provide an open platform to efficiently compute semantic distance, including tutorials and documentation (https://osf.io/gz4fc/). The online version of this article (10.3758/s13428-020-01453-w) contains supplementary material, which is available to authorized users.
登录
查看更多内容
影响因子：
3.6
作者：
Beaty, Roger E.;Silvia, Paul J.
通讯作者：
Silvia, Paul J.
影响因子：
2.6
作者：
Bossomaier, Terry;Harre, Mike;Snyder, Allan
通讯作者：
Snyder, Allan
影响因子：
3
作者：
Benedek, Mathias;Jauk, Emanuel;Sommer, Markus;Arendasy, Martin;Neubauer, Aljoscha C.
通讯作者：
Neubauer, Aljoscha C.
影响因子：
2.4
作者：
Beaty, Roger E.;Silvia, Paul J.;Benedek, Mathias
通讯作者：
Benedek, Mathias
影响因子：
3.6
作者：
Barbot, Baptiste;Besancon, Maud;Lubart, Todd
通讯作者：
Lubart, Todd