Evaluation for Actionable Change: A Data-Driven Approach

本提案是对促进评价方法研究和创新(PRIME)征集NSF 15-540的响应而提交的。PRIME方案旨在支持评价研究，特别强调探索创新方法，建立和扩大理论基础，并发展评价领域的能力和基础设施。越来越大的问责压力导致了对教育项目和实践的严格评估。然而，严格的评估，如随机对照试验(RCT)，费用昂贵，而且往往显示出很小的效果。即使是被广泛采用的数字学习平台的随机对照试验也可能显示令人失望的结果，这些结果对随后采用已经在教育领域根深蒂固的项目几乎没有影响。需要新的方法来估计效果，并指出改善已经采用的数字学习工具的结果的方法。随着平台目前的广泛使用，评估使用模式及其与结果的关系的新方法既可以评估最大效率，也可以提供提高效率的手段。这项研究将开发ST Math的评估，这是一个K-8数字学习平台，旨在通过提高学生对数学概念的理解和数学学习的动机来加强他们的数学能力。通过投资于评估创新以改进ST Math等数字学习平台，此类项目可以在每次迭代中以最大的效果惠及大量儿童。开发改进的自动化工具还可以提高数字学习平台的效率和效力。这项工作将通过北卡罗来纳州立大学(NC State)在教育评估、教育数据挖掘和评估方面的专业知识的研究人员与非营利性心理研究所(Mind)的程序开发人员合作完成。该项目将探索新的、非侵入性的方法，以确定ST Math数字学习环境的影响。它将推进使用过程数据进行形成性评估的分析基础，并通过促进自动识别学习的可教时刻来建立改进STEM教与学的算法。这项研究的变革潜力在于创造了新的跨学科方法，不仅可以用来评估影响，而且通过利用观察到的学生和教师的行为和教育数据挖掘技术，为改善STEM的教与学提供信息。具体地说，这项研究将探索检测、可视化和评估学生的新方法。解谜和选择谜题行为。督学将评估检测到的模式是否由学生驱动？未来的能力或可以用来预测他们的短期和长期表现。通过将学生和教师的行为模式与重要的学习和动机结果联系起来，研究人员或许能够向教师推荐有前途的行动，向开发人员推荐潜在的改进措施。这项工作不仅有可能改变ST Math平台的使用和成功，而且还有可能创造出可以改进并转移到其他平台的评估和实施的方法。

This proposal was submitted in response to the Promoting Research and Innovation in Methodologies for Evaluation (PRIME) solicitation NSF 15-540. The PRIME program seeks to support research on evaluation with special emphasis on exploring innovative approaches, building on and expanding the theoretical foundations, and growing the capacity and infrastructure of the evaluation field. Increasing pressures for accountability have resulted in a push for rigorous evaluation of educational programs and practice. Yet rigorous evaluations such as Randomized Control Trials (RCTs) are expensive and often show small effects. Even RCTs of widely-adopted digital learning platforms can show disappointing results and these results have little impact on subsequent adoptions of programs already entrenched in the educational landscape. New methods are needed to both estimate effects and to indicate ways of improving outcomes for already-adopted digital learning tools. With platforms currently in wide-scale use, novel approaches to assessing use patterns and their relations with outcomes can both evaluate maximal effectiveness and provide means for improved effectiveness. This research will develop an evaluation of ST Math, a K-8 digital learning platform designed to strengthen the mathematical competency of students through enhancing both their understanding of math concepts and their motivation for math learning. By investing in evaluation innovations to improve digital learning platforms such as ST Math, such programs could reach large numbers of children with maximal effectiveness with each iteration. Development of automated tools for improvement can also enhance both the efficiency and efficacy of the digital learning platforms. This work will be accomplished through a partnership between researchers at North Carolina State University (NC State) with expertise in educational evaluation, educational data mining, and assessment with program developers at the non-profit MIND Research Institute (MIND).This project will explore novel, and noninvasive, approaches for determining the impact of the ST Math digital learning environment. It will advance the analytical basis for formative assessment using process data and build algorithms that improve STEM teaching and learning by facilitating the automatic recognition of teachable moments for learning. The transformative potential of this research resides in the creation of new cross-disciplinary approaches that can be used to not only evaluate impact, but to inform improved teaching and learning in STEM, by leveraging observed behaviors of students and teachers and educational data mining techniques. Specifically, this research will explore novel methods for detecting, visualizing, and evaluating students? puzzle-solving and puzzle-selection behaviors. The PIs will assess whether the detected patterns are driven by students? incoming competence or can be used to predict their short and long-term performance. By linking student and teacher behavior patterns with important learning and motivational outcomes, the researchers may be able to recommend promising actions to teachers and potential refinements to developers. This work has the potential to not only transform the use and success of the ST Math platform, but to create methods that can be refined and transferred to the evaluation and implementation of other platforms.