Do Speech-Based Collaboration Analytics Generalize Across Task Contexts?
Do Speech-Based Collaboration Analytics Generalize Across Task Contexts?
复制标题
DOI：
10.1145/3506860.3506894
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Samuel L. Pugh;A. Rao;Angela E. B. Stewart;S. D’Mello
中科院分区：
文献类型：
--
作者：
Samuel L. Pugh;A. Rao;Angela E. B. Stewart;S. D’Mello
我们研究了基于语言的分析模型在两个协作问题解决（CPS）任务中的通用性：教育物理游戏和块编程挑战。我们分析了95个三合会（N=285）的数据集，他们使用视频会议在两个任务上进行了一个小时的合作。我们在自动语音识别成绩单上训练了有监督的自然语言处理分类器，以预测构建共享知识，谈判/协调和维护团队功能的人类编码CPS方面（技能）。我们测试了三种表示协作话语的方法：（1）深度迁移学习（使用BERT），（2）n-gram（单词/短语计数）和（3）单词类别（使用Linguistic Inquiry Word Count [LIWC]词典）。我们发现BERT和LIWC方法在任务之间进行了泛化，性能只有很小的下降（转移率为0.93，1表示完美转移），而n-gram的泛化能力有限（转移率为0.86），这表明对特定任务语言的过度拟合。我们讨论了我们的研究结果部署基于语言的协作分析在真实的教育环境中的影响。
We investigated the generalizability of language-based analytics models across two collaborative problem solving (CPS) tasks: an educational physics game and a block programming challenge. We analyzed a dataset of 95 triads (N=285) who used videoconferencing to collaborate on both tasks for an hour. We trained supervised natural language processing classifiers on automatic speech recognition transcripts to predict the human-coded CPS facets (skills) of constructing shared knowledge, negotiation / coordination, and maintaining team function. We tested three methods for representing collaborative discourse: (1) deep transfer learning (using BERT), (2) n-grams (counts of words/phrases), and (3) word categories (using the Linguistic Inquiry Word Count [LIWC] dictionary). We found that the BERT and LIWC methods generalized across tasks with only a small degradation in performance (Transfer Ratio of .93 with 1 indicating perfect transfer), while the n-grams had limited generalizability (Transfer Ratio of .86), suggesting overfitting to task-specific language. We discuss the implications of our findings for deploying language-based collaboration analytics in authentic educational environments.