AF: Small: Mechanism Design for the Classroom

AF：小：课堂的机制设计

• 批准号: 2229162
• 负责人: Jason Hartline
• 金额: $ 60万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2229162&HistoricalAwards=false
• 关键词: AF; Small; Mechanism; Design; Classroom

Mechanism design studies how the rules of a system can be designed so that good outcomes are obtained when individuals participating in the system are strategic. This project develops -- and initiates the theoretical study of -- a collection of mechanism design problems for the classroom. Specifically, it views the classroom as a computational system where some participants may manipulate the system to obtain better individual outcomes (i.e., the students) and some participants may be unreliable (i.e., the graders). The instructor aims to put in place policies with a number of natural objectives, e.g., optimizing learning outcomes, fairness of grading policies, and efficiency with respect to effort from participants (both students and graders). By understanding the classroom as an application domain for mechanism design, classroom outcomes can be improved. Moreover, a foundation for mechanism design that is grounded in practice can be established. This foundation may have an impact on other application domains for mechanism design, such as online markets.This project explores three main thrusts: fairness in heterogeneous grading, grading to optimize study incentives, and the design of student feedback mechanisms. Thrust 1: Randomizing questions from a large bank of questions is a popular cheating deterrent in online exams. When students are assigned questions with heterogeneous difficulties, however, fair assessment is not straightforward. Some students may be assigned easier questions than others and the simple averaging of scores will favor these more fortunate students. A performance benchmark for fair and accurate assessment is a student’s average grade on the full question bank. This project aims to (a) develop grading algorithms that perform well with respect to this benchmark for any assignment of questions to students, and (b) understand the impact of the structure of the assignment of questions to students on performance, i.e., exam design. Thrust 2: When students are assigned tasks, their level of effort depends on how their effort is graded. Effort can result in learning; however, it is not directly observed. For example, an article reading task might be assessed via reading comprehension questions where answers to these questions can be assessed, but the amount of effort of reading cannot be observed. The grading of knowledge can be understood in the paradigm of scoring rules. Scoring rules are a classical paradigm for incentivizing a forecaster to report a prediction about an unknown state. In the classroom context, student answers to questions about course material can be interpreted as a prediction of the correct answer. In this context, this project aims to develop a theory for the optimization of scoring rules, i.e., identifying scoring rules that incentivize the students to exert effort (that results in learning). Thrust 3: Feedback to students enables them to assess how their effort leads to outcomes that are relevant to them, such as learning or grades. For example, less precise feedback on grades could lead to more consistent effort because it would avoid students overreacting to spurious high or low grades (cf. overfitting in machine learning). Information design is a classical paradigm where a principal signals an agent about an unknown state to entice the agent to take a more favorable action for the principal. The project aims to understand the sequential provision of feedback to students as a problem of information design and to design good feedback mechanisms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机制设计研究如何设计系统的规则，以便当参与系统的个人具有战略性时获得良好的结果。 这个项目开发-并启动了理论研究-一系列的机制设计问题的课堂。具体来说，它将课堂视为一个计算系统，其中一些参与者可以操纵系统以获得更好的个人结果（即，学生）并且一些参与者可能是不可靠的（即，分级者）。 讲师旨在制定具有许多自然目标的政策，例如，优化学习成果，评分政策的公平性，以及参与者（学生和评分者）努力的效率。通过将课堂理解为机制设计的应用领域，可以提高课堂效果。 此外，还可以建立基于实践的机制设计基础。这个基础可能会影响到其他应用领域的机制设计，如在线市场。本项目探讨了三个主要的推力：在异构评分的公平性，评分，以优化学习激励，和学生反馈机制的设计。 要点1：从大量的问题中随机出题是在线考试中常见的作弊威慑手段。 然而，当学生被分配到不同难度的问题时，公平的评估并不简单。 一些学生可能会被分配比其他学生更容易的问题，简单的平均分数将有利于这些更幸运的学生。 公平和准确评估的表现基准是学生在完整问题库中的平均成绩。 本项目旨在（a）开发评分算法，该算法在任何向学生分配问题的基准方面表现良好，以及（B）了解向学生分配问题的结构对表现的影响，即，考试设计 重点2：当学生被分配任务时，他们的努力程度取决于他们的努力如何被评分。 努力可以导致学习;然而，它不是直接观察到的。 例如，可以通过阅读理解问题来评估文章阅读任务，其中可以评估这些问题的答案，但是不能观察阅读的努力量。 知识的分级可以用评分规则的范式来理解。 评分规则是激励预测者报告关于未知状态的预测的经典范例。 在课堂上，学生对课程材料问题的回答可以被解释为对正确答案的预测。 在此背景下，本项目旨在开发一种优化评分规则的理论，即，确定激励学生努力（导致学习）的评分规则。 重点3：对学生的反馈使他们能够评估他们的努力如何导致与他们相关的结果，如学习或成绩。 例如，对分数的反馈不太精确，可能会导致更一致的努力，因为它会避免学生对虚假的高或低分数反应过度（参见。机器学习中的过拟合）。 信息设计是一个经典的范例，其中委托人向代理人发出关于未知状态的信号，以诱使代理人采取对委托人更有利的行动。 该项目的目的是了解作为一个信息设计的问题，并设计良好的反馈机制的顺序提供反馈给学生。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Fair Grading Algorithms for Randomized Exams

随机考试的公平评分算法

• 发表时间: 2023
• 期刊: 4th Symposium on Foundations of Responsible Computing (FORC 2023
• 作者: Jiale Chen;Jason Hartline;Onno Zoeter
• 通讯作者: Onno Zoeter