Strategies for Deploying Unreliable AI Graders in High-Transparency High-Stakes Exams
Strategies for Deploying Unreliable AI Graders in High-Transparency High-Stakes Exams
复制标题
DOI：
10.1007/978-3-030-52237-7_2
复制
发表时间：
2020-06-09
期刊：
影响因子：
--
通讯作者：
Zilles C
中科院分区：
文献类型：
--
作者：
Azad S;Chen B;Fowler M;West M;Zilles C
我们描述了一个不完善的基于自然语言处理的自动简答评分系统在一门大招生大学入门课程的考试中的部署。我们将这种部署描述为高风险(问题在期中考试中相当于学生期末成绩的10%)和高透明度(问题在基于计算机的考试期间被交互评分，并向学生展示正确的答案，可以与他们的答案进行比较)。我们研究了两种技术，旨在缓解由于学生不正确地没有被不完美的人工智能评分器授予学分而导致的潜在学生不满。我们发现(1)提供多次尝试可以消除第一次尝试的假阴性，但代价是额外的假阳性；(2)没有从算法中获得学分的学生无法可靠地确定他们的答案是否被错误评分。
We describe the deployment of an imperfect NLP-based automatic short answer grading system on an exam in a large-enrollment introductory college course. We characterize this deployment as both high stakes (the questions were on an mid-term exam worth 10% of students’ final grade) and high transparency (the question was graded interactively during the computer-based exam and correct solutions were shown to students that could be compared to their answer). We study two techniques designed to mitigate the potential student dissatisfaction resulting from students incorrectly not granted credit by the imperfect AI grader. We find (1) that providing multiple attempts can eliminate first-attempt false negatives at the cost of additional false positives, and (2) that students not granted credit from the algorithm cannot reliably determine if their answer was mis-scored.
登录
查看更多内容
DOI：
10.1007/s40593-014-0026-8
发表时间：
2015-03-01
影响因子：
4.9
作者：
Burrows, Steven;Gurevych, Iryna;Stein, Benno
通讯作者：
Stein, Benno
DOI：
10.7717/peerj-cs.208
发表时间：
2019
期刊：
PeerJ. Computer science
影响因子：
--
作者：
Hussein MA;Hassan H;Nassef M
通讯作者：
Nassef M
DOI：
10.1023/a:1025779619903
发表时间：
2003-11-01
期刊：
COMPUTERS AND THE HUMANITIES
影响因子：
--
作者：
Leacock, C;Chodorow, M
通讯作者：
Chodorow, M
影响因子：
6
作者：
Sam, Amir H.;Field, Samantha M.;Meeran, Karim
通讯作者：
Meeran, Karim