AitF: Mechanism Design and Machine Learning for Peer Grading

本专题探讨同侪评等技术的设计与分析。 同行评分系统是一个在线工具，它收集学生提交的材料，向学生和评分员分配审查任务，并汇总审查结果，以生成对提交材料和同行审查的评估。 PI已经开发了一个原型系统，并收集了初步证据，表明同行评审具有重要的潜在好处：1。 通过回顾学习：学生从对其他学生作业的批判性评估中学习。 在西北大学的PI原型中，60%的学生报告说同伴互评帮助他们学习课程材料，55%的学生报告说同伴互评帮助他们自己准备更好的家庭作业解决方案。 减少评分人员：同行评分减少了课程工作人员的评分负担，并允许更大的班级进行有效的教学。 这一点目前尤其重要，因为对计算机科学课程的兴趣比教学资源增长得更快。 在西北大学的PI原型中，课程工作人员对学生提交的1/5进行评分。 反馈的准确性：减少教师评分，使及时反馈给学生。 在西北大学的PI原型中，同行评审在三天内完成，对提交材料和同行评审的最终评估在五天内完成。 在引入同行评议之前，评估需要一到两周的时间。同行评分系统由三个主要部分组成：1. 评论匹配算法确定哪些同伴应该评论哪些提交以及哪些提交应该由教师评论。 提交评分算法汇总同行和提交的评论，并为提交分配等级。3. 评论评分算法将同行评论与教师评论进行比较，并为同行评论分配分数。 如果没有这种算法，同行可能不会努力提供高质量的评论，评论既不会准确评分，也不会对同行有益。这些算法的细节对于同行评审系统的正常工作至关重要。 该项目的主要研究工作是确定用于每个组件的算法。 复习匹配算法影响后续评分算法的准确性和教师的评分负荷。 提交分级算法确定哪些同行评审是准确的，哪些是不准确的，并使用这种理解来为代表提交质量的提交分配等级。 评论评分算法激励同行付出足够的努力来确定提交是好还是坏，并且它被校准，以便好评论和差评论获得适当的评论等级。 PI已经实现了这些算法的原型，作为同行评分系统的一部分，该系统已在西北大学计算机科学课程中原型化。 然而，可能的算法空间很大，PI在原型上的工作尚未确定联合收割机的算法，以提供最好的教育成果。 该项目的一个主要重点将是提高对哪些算法导致最佳教育成果的理解。算法和机器学习方面的理论工作为该项目研究同伴评分系统的良好算法提供了起点。 该项目的一个关键奋进是将这些理论算法翻译并应用于同行评分领域。 例如，适当的评分规则是对同行评审进行评分的自然方法。 然而，PI原型实现的测试运行表明，这些规则在实践中可能并不那么好。 理论上需要新的模型和算法，这些新的算法需要在实践中工作。

This project explores the design and analysis of peer grading technology. A peer grading system is an online tool that collects student submissions, assigns review tasks to the students and graders, and aggregates reviews to produce assessments of both the submissions and the peer reviews. The PIs have developed a prototype system and have collected preliminary evidence that suggests that peer review has important potential benefits:1. Learning by reviewing: Students learn from critical assessment of other students' work. In the PIs' prototype at Northwestern, 60% of the students reported that peer review helped them learn course material and 55% of the students reported that peer review helped them to prepare better homework solutions themselves.2. Reduced grading staff: Peer grading reduces the grading load on course staff and allows for effective teaching with larger classes. This is especially important currently, as interest in computer science classes increases at a faster pace than teaching resources. In the PIs' prototype at Northwestern, the course staff graded 1/5 of the student submissions.3. Promptness of feedback: Reduced teacher grading enables prompt feedback to students. In the PIs' prototype at Northwestern, peer reviews were available within three days and final assessment of both the submission and peer reviews were available within five days. Prior to introducing peer review, assessments took one to two weeks.A peer grading system is comprised of three main components:1. The review matching algorithm determines which peers should review which submissions and which submissions should be reviewed by the teacher.2. The submission grading algorithm aggregates the reviews of the peers and the submissions and assigns grades to the submissions.3. The review grading algorithm compares the peer reviews with the teacher reviews and assigns grades to the peer reviews. Without this algorithm, peers may not put effort into providing quality reviews, and the reviews will be neither accurate for grading nor beneficial for the peer.The details of these algorithms are crucial for the proper working of the peer review system. A main research effort of this project is to identify the algorithms to use for each of these components. The review matching algorithm affects the accuracy of the subsequent grading algorithms and the grading load of the teacher. The submission grading algorithm determines which peer reviews are accurate and which are inaccurate and uses this understanding to assign grades to the submissions that are representative of the submission quality. The review grading algorithm incentivizes the peers to put in sufficient effort to determine whether a submission is good or bad and it is calibrated so that good reviews and bad reviews get the appropriate review grades. The PIs have implemented prototypes of these algorithms as part of a peer grading system that has been prototyped in Northwestern computer science classes. However, the space of possible algorithms is large and the PIs' work on the prototype has yet to determine the algorithms that combine to give the best education outcomes. A main focus of this project will be improving the understanding of which algorithms lead to the best education outcomes.Theoretical work in algorithms and machine learning provides a starting point for the project's study of good algorithms for peer grading systems. A key endeavor of the project is translating and applying these theoretical algorithms to the peer grading domain. As one example, proper scoring rules are a natural approach for grading the peer reviews. However, test runs of the PIs' prototype implementation suggest that these rules might not be so good in practice. Both new models and algorithms are needed in theory, and these new algorithms need to work in practice.