AitF: Mechanism Design and Machine Learning for Peer Grading

AitF：同行评分的机制设计和机器学习

• 批准号: 1733860
• 负责人: Jason Hartline
• 金额: $ 70万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1733860&HistoricalAwards=false
• 关键词: AitF; Mechanism; Design; Machine; Learning

This project explores the design and analysis of peer grading technology. A peer grading system is an online tool that collects student submissions, assigns review tasks to the students and graders, and aggregates reviews to produce assessments of both the submissions and the peer reviews. The PIs have developed a prototype system and have collected preliminary evidence that suggests that peer review has important potential benefits:1. Learning by reviewing: Students learn from critical assessment of other students' work. In the PIs' prototype at Northwestern, 60% of the students reported that peer review helped them learn course material and 55% of the students reported that peer review helped them to prepare better homework solutions themselves.2. Reduced grading staff: Peer grading reduces the grading load on course staff and allows for effective teaching with larger classes. This is especially important currently, as interest in computer science classes increases at a faster pace than teaching resources. In the PIs' prototype at Northwestern, the course staff graded 1/5 of the student submissions.3. Promptness of feedback: Reduced teacher grading enables prompt feedback to students. In the PIs' prototype at Northwestern, peer reviews were available within three days and final assessment of both the submission and peer reviews were available within five days. Prior to introducing peer review, assessments took one to two weeks.A peer grading system is comprised of three main components:1. The review matching algorithm determines which peers should review which submissions and which submissions should be reviewed by the teacher.2. The submission grading algorithm aggregates the reviews of the peers and the submissions and assigns grades to the submissions.3. The review grading algorithm compares the peer reviews with the teacher reviews and assigns grades to the peer reviews. Without this algorithm, peers may not put effort into providing quality reviews, and the reviews will be neither accurate for grading nor beneficial for the peer.The details of these algorithms are crucial for the proper working of the peer review system. A main research effort of this project is to identify the algorithms to use for each of these components. The review matching algorithm affects the accuracy of the subsequent grading algorithms and the grading load of the teacher. The submission grading algorithm determines which peer reviews are accurate and which are inaccurate and uses this understanding to assign grades to the submissions that are representative of the submission quality. The review grading algorithm incentivizes the peers to put in sufficient effort to determine whether a submission is good or bad and it is calibrated so that good reviews and bad reviews get the appropriate review grades. The PIs have implemented prototypes of these algorithms as part of a peer grading system that has been prototyped in Northwestern computer science classes. However, the space of possible algorithms is large and the PIs' work on the prototype has yet to determine the algorithms that combine to give the best education outcomes. A main focus of this project will be improving the understanding of which algorithms lead to the best education outcomes.Theoretical work in algorithms and machine learning provides a starting point for the project's study of good algorithms for peer grading systems. A key endeavor of the project is translating and applying these theoretical algorithms to the peer grading domain. As one example, proper scoring rules are a natural approach for grading the peer reviews. However, test runs of the PIs' prototype implementation suggest that these rules might not be so good in practice. Both new models and algorithms are needed in theory, and these new algorithms need to work in practice.

该项目探讨了同行评分技术的设计和分析。 同行评分系统是一种在线工具，可以收集学生提交的内容，将审查任务分配给学生和分级者，并汇总评论，以进行对提交和同行评审的评估。 PI已经开发了一个原型系统，并收集了初步证据，表明同行评审具有重要的潜在好处：1。 通过审查学习：学生从对其他学生的工作的批判性评估中学习。 在西北地区的PIS原型中，有60％的学生报告说，同行评审帮助他们学习了课程材料，而55％的学生报告说，同行评审帮助他们自己准备了更好的家庭作业解决方案2。 减少评分人员：同行评分减少了课程人员的评分负载，并可以在更大的课程中有效教学。 目前，这一点尤其重要，因为对计算机科学课程的兴趣比教学资源更快。 在西北地区的PIS原型中，课程工作人员分级为学生提交的1/5。3。 反馈的及时性：减少的教师评分可以使学生及时反馈。 在Northwestern的PIS原型中，在三天之内进行了同行评审，并且对提交和同行评审的最终评估可在五天内进行。 在引入同行评审之前，评估需要一到两周。同行分级系统由三个主要组成部分组成：1。 审核匹配算法确定哪些同行应审查哪些提交以及应由教师审查哪些提交。2。 提交评分算法汇总了对同行的评论以及提交和分配成绩的评论。3。 评论分级算法将同行评论与教师评论进行比较，并将成绩分配给同行评审。 没有这种算法，同伴可能不会付出努力来提供质量审查，并且评论既不准确地对等级，也不是有益的。这些算法的详细信息对于同行评审系统的适当工作至关重要。 该项目的主要研究工作是确定要用于每个组件的算法。 评论匹配算法会影响随后的分级算法的准确性和教师的分级负载。 提交评分算法确定哪些同行评审是准确的，哪些是不准确的，并利用此理解将成绩分配给代表提交质量的提交。 评论分级算法激发了同龄人付出足够的努力来确定提交是好还是坏，并且经过校准，以使良好的评论和不良评论获得适当的评论等级。 PI是在西北计算机科学课上进行了原型的同行评分系统的一部分，实施了这些算法的原型。 但是，可能的算法空间很大，PIS在原型上的工作尚未确定结合以提供最佳教育成果的算法。 该项目的一个主要重点将是提高人们对哪种算法导致最佳教育成果的理解。算法和机器学习的理论工作为该项目研究对同行分级系统的良好算法的研究提供了一个起点。 该项目的关键努力是将这些理论算法转换为对等级域。 例如，适当的评分规则是对同行评审进行评分的自然方法。 但是，PIS原型实现的测试运行表明，这些规则在实践中可能不是那么好。 从理论上讲，新模型和算法都是必需的，这些新算法在实践中需要工作。

项目成果

Optimization of Scoring Rules

评分规则优化

• DOI: 10.1145/3490486.3538338
• 发表时间: 2022
• 期刊: ACM Conference on Economics and Computation
• 作者: Li, Yingkai;Hartline, Jason D.;Shan, Liren;Wu, Yifan
• 通讯作者: Wu, Yifan

Practical Methods for Semi-automated Peer Grading in a Classroom Setting

课堂环境中半自动同伴评分的实用方法

• DOI: 10.1145/3340631.3394878
• 发表时间: 2020
• 期刊: Adaptation and Personalization
• 作者: Yuan, Zheng;Downey, Doug
• 通讯作者: Downey, Doug