Evaluation for Actionable Change: A Data-Driven Approach

评估可行的变革：数据驱动的方法

• 批准号: 1544273
• 负责人: Paola Sztajn
• 金额: $ 79.98万
• 依托单位: North Carolina State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-01-01 至 2021-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1544273&HistoricalAwards=false
• 关键词: Evaluation; Actionable; Change; Data; Driven

This proposal was submitted in response to the Promoting Research and Innovation in Methodologies for Evaluation (PRIME) solicitation NSF 15-540. The PRIME program seeks to support research on evaluation with special emphasis on exploring innovative approaches, building on and expanding the theoretical foundations, and growing the capacity and infrastructure of the evaluation field. Increasing pressures for accountability have resulted in a push for rigorous evaluation of educational programs and practice. Yet rigorous evaluations such as Randomized Control Trials (RCTs) are expensive and often show small effects. Even RCTs of widely-adopted digital learning platforms can show disappointing results and these results have little impact on subsequent adoptions of programs already entrenched in the educational landscape. New methods are needed to both estimate effects and to indicate ways of improving outcomes for already-adopted digital learning tools. With platforms currently in wide-scale use, novel approaches to assessing use patterns and their relations with outcomes can both evaluate maximal effectiveness and provide means for improved effectiveness. This research will develop an evaluation of ST Math, a K-8 digital learning platform designed to strengthen the mathematical competency of students through enhancing both their understanding of math concepts and their motivation for math learning. By investing in evaluation innovations to improve digital learning platforms such as ST Math, such programs could reach large numbers of children with maximal effectiveness with each iteration. Development of automated tools for improvement can also enhance both the efficiency and efficacy of the digital learning platforms. This work will be accomplished through a partnership between researchers at North Carolina State University (NC State) with expertise in educational evaluation, educational data mining, and assessment with program developers at the non-profit MIND Research Institute (MIND).This project will explore novel, and noninvasive, approaches for determining the impact of the ST Math digital learning environment. It will advance the analytical basis for formative assessment using process data and build algorithms that improve STEM teaching and learning by facilitating the automatic recognition of teachable moments for learning. The transformative potential of this research resides in the creation of new cross-disciplinary approaches that can be used to not only evaluate impact, but to inform improved teaching and learning in STEM, by leveraging observed behaviors of students and teachers and educational data mining techniques. Specifically, this research will explore novel methods for detecting, visualizing, and evaluating students? puzzle-solving and puzzle-selection behaviors. The PIs will assess whether the detected patterns are driven by students? incoming competence or can be used to predict their short and long-term performance. By linking student and teacher behavior patterns with important learning and motivational outcomes, the researchers may be able to recommend promising actions to teachers and potential refinements to developers. This work has the potential to not only transform the use and success of the ST Math platform, but to create methods that can be refined and transferred to the evaluation and implementation of other platforms.

该提案是为了响应促进评价方法研究和创新（PRIME）的NSF 15-540征集而提交的。PRIME计划旨在支持评价研究，特别强调探索创新方法，建立和扩大理论基础，并提高评价领域的能力和基础设施。问责压力的增加导致了对教育项目和实践的严格评估。然而，严格的评估，如随机对照试验（RCT）是昂贵的，往往显示小的效果。即使是广泛采用的数字学习平台的随机对照试验也可能显示出令人失望的结果，这些结果对已经在教育领域根深蒂固的项目的后续采用几乎没有影响。需要新的方法来估计效果，并指出改善已采用的数字学习工具的结果的方法。随着平台目前的广泛使用，评估使用模式及其与结果关系的新方法既可以评估最大有效性，又可以提供提高有效性的手段。这项研究将开发对ST Math的评估，ST Math是一个K-8数字学习平台，旨在通过增强学生对数学概念的理解和数学学习的动机来增强他们的数学能力。通过投资评估创新来改善ST Math等数字学习平台，这些项目可以在每次迭代中以最大的效率覆盖大量儿童。开发自动化改进工具也可以提高数字学习平台的效率和功效。这项工作将通过北卡罗来纳州州立大学（NC State）的研究人员与非营利性MIND研究所（MIND）的程序开发人员在教育评估、教育数据挖掘和评估方面的专业知识合作来完成。该项目将探索新的、非侵入性的方法来确定ST数学数字学习环境的影响。它将使用过程数据推进形成性评估的分析基础，并建立算法，通过促进自动识别学习的可教时刻来改善STEM教学和学习。这项研究的变革潜力在于创造新的跨学科方法，不仅可用于评估影响，还可通过利用观察到的学生和教师的行为以及教育数据挖掘技术来改善STEM的教学和学习。具体而言，本研究将探索新的方法来检测，可视化，并评估学生？解谜和选谜行为。PI将评估检测到的模式是否由学生驱动？未来的能力，或可以用来预测他们的短期和长期的表现。通过将学生和教师的行为模式与重要的学习和动机结果联系起来，研究人员可以向教师推荐有希望的行动，并向开发人员推荐潜在的改进措施。这项工作不仅有可能改变ST Math平台的使用和成功，而且还可以创建可以改进并转移到其他平台的评估和实施的方法。