Statistical Consequences of using Multi-armed Bandits to Conduct Adaptive Educational Experiments
Statistical Consequences of using Multi-armed Bandits to Conduct Adaptive Educational Experiments
复制标题
使用多臂老虎机进行适应性教育实验的统计结果
DOI：
10.5281/zenodo.3554749
复制
发表时间：
2019
影响因子：
3.7
通讯作者：
J. Williams
中科院分区：
文献类型：
--
作者：
Anna N. Rafferty;Huiji Ying;J. Williams
随机实验可以为改进教育技术提供关键见解，但许多学生可能会在这些实验中遇到与较差学习结果相关的条件。多臂强盗（MAB）算法可以通过在实验运行时积累实验证据并修改实验设计来为更大比例的未来学生分配更有帮助的条件来解决这个问题。使用模拟，我们探讨了使用MAB算法进行实验设计的统计影响，重点是从实验中获取统计可靠的信息和学生的利益之间的权衡。我们认为学生行为模式的时间偏差可能会影响MAB实验的结果，并从以前的10个教育实验的模型数据，以证明MAB分配的潜在影响。结果表明，MAB实验可以导致更高的平均收益，学生比传统的实验设计，虽然至少有两倍的参与者需要可接受的统计能力。对MAB算法使用乐观的先验分布在一定程度上减轻了功率的损失，而不会显着减少对学生的好处。此外，较长的实验与MAB分配仍然分配更少的学生到一个不太有效的条件比典型的做法，较短的实验，然后选择一个条件，为所有未来的学生。然而，MAB分配确实增加了假阳性率，特别是当学生进入实验时存在时间偏差时。因此，在学生可以选择何时参加实验的情况下，解释MAB作业的结果时必须谨慎。总的来说，在学生特征不随时间变化的情况下，MAB实验设计对学生有益，并且在大样本量的情况下，可以有效地可靠地确定两种不同条件中的哪一种更好。
Randomized experiments can provide key insights for improving educational technologies, but many students may experience conditions associated with inferior learning outcomes in these experiments. Multiarmed bandit (MAB) algorithms can address this issue by accumulating evidence from the experiment as it runs and modifying the experimental design to assign more helpful conditions to a greater proportion of future students. Using simulations, we explore the statistical impact of using MAB algorithms for experiment design, focusing on the tradeoff between acquiring statistically reliable information from the experiment and benefits to students. We consider how temporal biases in patterns of student behavior may impact the results of MAB experiments, and model data from ten previous educational experiments to demonstrate potential impacts of MAB assignment. Results suggest that MAB experiments can lead to much higher average benefits to students than traditional experimental designs, although at least twice as many participants are needed for acceptable statistical power. Using an optimistic prior distribution for the MAB algorithm mitigates the loss in power to some extent, without significantly reducing benefits to students. Additionally, longer experiments with MAB assignment still assign fewer students to a less effective condition than typical practice of a shorter experiment followed by choosing one condition for all future students. Yet, MAB assignment does increase false positive rates, especially if there are temporal biases in when students enter the experiment. Caution must thus be used when interpreting results from MAB assignment in cases where students can choose when to participate in the experiment. Overall, in scenarios where student characteristics do not vary over time, MAB experimental designs can be beneficial for students and effective for reliably determining which of two differing conditions is better given large sample sizes.
将自适应性与智能辅导系统的进度排序相结合
DOI：
10.1145/3231644.3231672
发表时间：
2018
期刊：
Learning at Scale 2018
影响因子：
--
作者：
Mu, Tong;Wang, Shuhan;Andersen, Erik;Brunskill, Emma
通讯作者：
Brunskill, Emma