Evaluation for Actionable Change: A Data-Driven Approach

该提案是根据促进评估方法的研究和创新（PRIME）招标NSF 15-540提交的。PRIME计划旨在支持评估研究，特别强调探索创新方法，建立和扩展理论基础，以及发展评估领域的能力和基础设施。越来越大的问责压力导致了对教育项目和实践的严格评估。然而，像随机对照试验（RCTs）这样的严格评估是昂贵的，而且往往显示出很小的效果。即使是广泛采用的数字学习平台的随机对照试验也会显示出令人失望的结果，这些结果对随后采用已经在教育领域根深蒂固的项目几乎没有影响。需要新的方法来评估效果，并指出改善已经采用的数字学习工具的结果的方法。随着平台的广泛使用，评估使用模式及其与结果的关系的新方法既可以评估最大有效性，也可以为提高有效性提供手段。本研究将开发ST Math的评估，这是一个K-8数字学习平台，旨在通过提高学生对数学概念的理解和数学学习的动机来加强学生的数学能力。通过投资于评估创新来改进ST Math等数字学习平台，这些项目可以在每次迭代中以最大的效率覆盖大量儿童。开发用于改进的自动化工具也可以提高数字学习平台的效率和功效。这项工作将通过北卡罗莱纳州立大学（NC State）的研究人员与非营利性MIND研究所（MIND）的程序开发人员在教育评估、教育数据挖掘和评估方面的专业知识之间的合作来完成。该项目将探索新的、无创的方法来确定ST数学数字学习环境的影响。它将利用过程数据推进形成性评估的分析基础，并构建算法，通过促进自动识别可教的学习时刻来改善STEM教与学。这项研究的变革潜力在于创造新的跨学科方法，不仅可以用来评估影响，还可以通过利用观察到的学生和教师的行为以及教育数据挖掘技术来改善STEM的教学和学习。具体来说，这项研究将探索新的方法来检测，可视化和评估学生？谜题解决和谜题选择行为。pi将评估检测到的模式是否由学生驱动？未来的能力或可以用来预测他们的短期和长期的表现。通过将学生和教师的行为模式与重要的学习和激励结果联系起来，研究人员可能能够向教师推荐有希望的行动，并向开发人员推荐潜在的改进。这项工作不仅有可能改变ST Math平台的使用和成功，而且有可能创造出可以改进并转移到其他平台的评估和实施的方法。

This proposal was submitted in response to the Promoting Research and Innovation in Methodologies for Evaluation (PRIME) solicitation NSF 15-540. The PRIME program seeks to support research on evaluation with special emphasis on exploring innovative approaches, building on and expanding the theoretical foundations, and growing the capacity and infrastructure of the evaluation field. Increasing pressures for accountability have resulted in a push for rigorous evaluation of educational programs and practice. Yet rigorous evaluations such as Randomized Control Trials (RCTs) are expensive and often show small effects. Even RCTs of widely-adopted digital learning platforms can show disappointing results and these results have little impact on subsequent adoptions of programs already entrenched in the educational landscape. New methods are needed to both estimate effects and to indicate ways of improving outcomes for already-adopted digital learning tools. With platforms currently in wide-scale use, novel approaches to assessing use patterns and their relations with outcomes can both evaluate maximal effectiveness and provide means for improved effectiveness. This research will develop an evaluation of ST Math, a K-8 digital learning platform designed to strengthen the mathematical competency of students through enhancing both their understanding of math concepts and their motivation for math learning. By investing in evaluation innovations to improve digital learning platforms such as ST Math, such programs could reach large numbers of children with maximal effectiveness with each iteration. Development of automated tools for improvement can also enhance both the efficiency and efficacy of the digital learning platforms. This work will be accomplished through a partnership between researchers at North Carolina State University (NC State) with expertise in educational evaluation, educational data mining, and assessment with program developers at the non-profit MIND Research Institute (MIND).This project will explore novel, and noninvasive, approaches for determining the impact of the ST Math digital learning environment. It will advance the analytical basis for formative assessment using process data and build algorithms that improve STEM teaching and learning by facilitating the automatic recognition of teachable moments for learning. The transformative potential of this research resides in the creation of new cross-disciplinary approaches that can be used to not only evaluate impact, but to inform improved teaching and learning in STEM, by leveraging observed behaviors of students and teachers and educational data mining techniques. Specifically, this research will explore novel methods for detecting, visualizing, and evaluating students? puzzle-solving and puzzle-selection behaviors. The PIs will assess whether the detected patterns are driven by students? incoming competence or can be used to predict their short and long-term performance. By linking student and teacher behavior patterns with important learning and motivational outcomes, the researchers may be able to recommend promising actions to teachers and potential refinements to developers. This work has the potential to not only transform the use and success of the ST Math platform, but to create methods that can be refined and transferred to the evaluation and implementation of other platforms.