AitF: Mechanism Design and Machine Learning for Peer Grading

本课题对同行评分技术的设计与分析进行了探索。同行评分系统是一个在线工具，它收集学生提交的材料，向学生和评分者分配审查任务，并汇总评论以产生对提交材料和同行审查的评估。PIs开发了一个原型系统，并收集了初步证据，表明同行评议具有重要的潜在好处：1.在复习中学习：学生从对其他学生作业的批判性评估中学习。在西北大学PIS的原型中，60%的学生报告同行评议帮助他们学习课程材料，55%的学生报告同行评议帮助他们自己准备更好的家庭作业解决方案。减少评分人员：同伴评分减轻了课程人员的评分负担，并允许在大班情况下进行有效教学。这一点在当前尤为重要，因为人们对计算机科学课程的兴趣增长速度快于教学资源。在西北大学PI的原型中，课程工作人员给五分之一的学生提交的材料打分。反馈迅速：降低教师评分使学生能够迅速得到反馈。在西北大学的PIS原型中，同行评审在三天内完成，提交和同行评审的最终评估在五天内完成。在引入同行评议之前，评估需要一到两周的时间。同行评分系统由三个主要组成部分组成：1.评审匹配算法确定哪些同行应该评审哪些提交的作品，哪些提交的作品应该由教师评审。提交评分算法将同行的评论和提交进行聚合，并对提交进行评分。点评评分算法将同行点评与教师点评进行比较，并对同行点评进行评分。如果没有该算法，同行可能不会努力提供高质量的评论，评论既不能准确评分，也不能对同行有利，这些算法的细节对于同行评审系统的正常工作至关重要。该项目的一个主要研究工作是确定用于每个组件的算法。复习匹配算法影响到后续评分算法的准确性和教师的评分负担。提交评分算法确定哪些同行评审是准确的，哪些是不准确的，并使用这种理解来对代表提交质量的提交进行评级。评论评分算法激励同行投入足够的努力来确定提交的文章是好是坏，并对其进行校准，以便好评论和坏评论获得适当的评论评分。PI已经实现了这些算法的原型，作为同行评分系统的一部分，该系统已在西北大学的计算机科学课程中原型。然而，可能的算法空间很大，PI在原型上的工作还没有确定结合在一起给出最佳教育结果的算法。这个项目的一个主要焦点将是提高对哪些算法能够产生最好的教育结果的理解。算法和机器学习方面的理论工作为该项目为同行评分系统研究良好的算法提供了一个起点。该项目的一个关键工作是将这些理论算法翻译并应用到同行评分领域。例如，适当的评分规则是对同行评议进行评分的自然方法。然而，PI原型实现的测试运行表明，这些规则在实践中可能并不是那么好。理论上需要新的模型和算法，而这些新的算法需要在实践中发挥作用。

This project explores the design and analysis of peer grading technology. A peer grading system is an online tool that collects student submissions, assigns review tasks to the students and graders, and aggregates reviews to produce assessments of both the submissions and the peer reviews. The PIs have developed a prototype system and have collected preliminary evidence that suggests that peer review has important potential benefits:1. Learning by reviewing: Students learn from critical assessment of other students' work. In the PIs' prototype at Northwestern, 60% of the students reported that peer review helped them learn course material and 55% of the students reported that peer review helped them to prepare better homework solutions themselves.2. Reduced grading staff: Peer grading reduces the grading load on course staff and allows for effective teaching with larger classes. This is especially important currently, as interest in computer science classes increases at a faster pace than teaching resources. In the PIs' prototype at Northwestern, the course staff graded 1/5 of the student submissions.3. Promptness of feedback: Reduced teacher grading enables prompt feedback to students. In the PIs' prototype at Northwestern, peer reviews were available within three days and final assessment of both the submission and peer reviews were available within five days. Prior to introducing peer review, assessments took one to two weeks.A peer grading system is comprised of three main components:1. The review matching algorithm determines which peers should review which submissions and which submissions should be reviewed by the teacher.2. The submission grading algorithm aggregates the reviews of the peers and the submissions and assigns grades to the submissions.3. The review grading algorithm compares the peer reviews with the teacher reviews and assigns grades to the peer reviews. Without this algorithm, peers may not put effort into providing quality reviews, and the reviews will be neither accurate for grading nor beneficial for the peer.The details of these algorithms are crucial for the proper working of the peer review system. A main research effort of this project is to identify the algorithms to use for each of these components. The review matching algorithm affects the accuracy of the subsequent grading algorithms and the grading load of the teacher. The submission grading algorithm determines which peer reviews are accurate and which are inaccurate and uses this understanding to assign grades to the submissions that are representative of the submission quality. The review grading algorithm incentivizes the peers to put in sufficient effort to determine whether a submission is good or bad and it is calibrated so that good reviews and bad reviews get the appropriate review grades. The PIs have implemented prototypes of these algorithms as part of a peer grading system that has been prototyped in Northwestern computer science classes. However, the space of possible algorithms is large and the PIs' work on the prototype has yet to determine the algorithms that combine to give the best education outcomes. A main focus of this project will be improving the understanding of which algorithms lead to the best education outcomes.Theoretical work in algorithms and machine learning provides a starting point for the project's study of good algorithms for peer grading systems. A key endeavor of the project is translating and applying these theoretical algorithms to the peer grading domain. As one example, proper scoring rules are a natural approach for grading the peer reviews. However, test runs of the PIs' prototype implementation suggest that these rules might not be so good in practice. Both new models and algorithms are needed in theory, and these new algorithms need to work in practice.