Making Math Tutors More Engaging and Effective through Interaction Design Patterns and Educational Data Mining

该项目开发并验证了交互设计模式、可大规模应用的高质量设计解决方案的结构化定义，这些解决方案可用于设计更有效、更吸引人的在线数学问题。本研究是在ASSISTments的背景下进行的，ASSISTments是全国中学生使用的免费在线数学软件。这些模式的设计使用了数千名学生之前使用过的2万个数学问题的数据（产生了数百万个数据点），并通过一组40个小型随机对照试验进行了验证，这些试验是通过自动化实验在美国全国各地的课堂上进行的。该项目是教师学院、哥伦比亚大学、卡内基梅隆大学和伍斯特理工学院的合作项目。该项目有两种贡献。第一个是基本的发现，即哪种类型的内容可以最好地学习和参与在线问题解决。第二是一套在线学习设计指南，教师和其他内容创作者可以应用这些指南来制作具有教育效果和吸引力的在线数学问题。项目通过以下步骤来实现这些目标。首先，他们正在研究现有ASSISTments数学问题中存在的设计特征，通过在一小部分问题上手工标记设计特征，然后使用教育数据挖掘来大规模复制手工标签。这些设计特性包括与界面设计、领域内容和教学策略相关的特性。然后，他们将先前开发和验证的学生学习、参与和影响的自动检测器应用于学生在ASSISTments中解决数学问题的日志文件。他们使用关联规则挖掘来确定设计特征的哪些组合会带来更好的学习、参与和影响，并在这些发现的基础上开发交互设计模式，从而传达结合这些数据特征的有效解决方案。设计模式通过一组40个小型随机对照试验通过自动化实验进行验证，其中40个数学问题使用设计模式得到改进。每个改进的问题都在200名学生（从目前使用ASSISTments作为常规课程一部分的学生中抽取）的随机样本中进行研究，这些学生将问题作为常规课堂活动的一部分。他们统计评估这些修改后的问题对学习和参与的影响，使用自动检测器的输出作为依赖措施。该项目提高了ASSISTments系统中数学问题的有效性和参与性，并确定了可用于改进ASSISTments中所有内容的设计模式。每年有50,000名学生使用ASSISTments系统，其中包括大量来自传统上代表性不足的人群的学生。更广泛地说，拟议的项目正在为创建更有效和更吸引人的在线学习提供一种可概括和精确的方法。所开发的设计模式可能有助于改进美国数学教育中越来越多使用的在线解决问题系统的设计。

This project develops and validates interaction design patterns, structured definitions of high-quality design solutions that can be applied at scale, that can be used to design more effective and engaging online mathematics problems. This research is conducted in the context of ASSISTments, free online mathematics software used by middle school students nationwide. The patterns are designed using data from 20,000 mathematics problems previously used by thousands of students (generating millions of data points), and validated through a set of forty small-scale randomized controlled trials conducted via automated experimentation, conducted online in American classrooms nationwide. The project is a partnership among Teachers College, Columbia University, Carnegie Mellon University, and Worcester Polytechnic Institute. The project makes two types of contributions. The first is are basic discoveries as to which types of content lead to the best learning and engagement in online problem-solving. The second is a set of guidelines for the design of online learning that can be applied by teachers and other content creators to produce educationally effective and engaging online mathematics problems.The project is accomplishing these goals using the following procedure. First, they are studying the design features present in existing ASSISTments mathematics problems, by hand-labeling design features on a small sub-set of problems and then using educational data mining to replicate the hand-labels at scale. These design features include features relevant to interface design, domain content, and pedagogical strategies. They then apply previously developed and validated automated detectors of student learning, engagement, and affect to log files of students solving mathematics problems in ASSISTments. They use association rule mining to determine which combinations of design features lead to better learning, engagement, and affect, and build on these findings to develop interaction design patterns that communicate effective solutions which combine these data features. The design patterns are validated through a set of forty small-scale randomized controlled trials conducted via automated experimentation, where forty mathematics problems are improved using the design patterns. Each improved problem is studied in a random sample of 200 students (drawn from the full population of students currently using ASSISTments as part of their regular curriculum), who receive the problem as part of their regular classroom activities. They statistically assess the impact of these modified problems on learning and engagement using the outputs of automated detectors as dependent measures. This project results in increasing the effectiveness and engagingness of the mathematics problems in the ASSISTments system, and identifies design patterns that can be applied to improve all of the content in ASSISTments. 50,000 students a year use the ASSISTments system, including large numbers of students from traditionally under-represented populations. More broadly, the proposed project is producin a generalizable and precise approach for the creation of more effective and engaging online learning. The design patterns developed are likely to be useful for improving the design of the range of online problem-solving systems used increasingly in American mathematics education.