Contextualized Evaluation Framework for Advanced STEM MOOCs

促进评估方法研究和创新(PRIME)计划支持评估研究，特别强调：(1)探索确定STEM教育项目和计划的影响和有用性的创新方法；(2)建立和扩展评估STEM教育和劳动力发展倡议的理论基础，包括翻译和调整其他领域的方法；以及(3)提高评估领域的能力和基础设施。普渡大学的PI和共同PI将进行一项重要的研究，旨在创建一种评估STEM MOOC的新范式，该范式超越了对点击流数据的分析，并可用于基于证据的决策。这一新范式的目标是通过考虑学习者特征、课程特征和利益相关者目标来实现评估指标的情境化。这一新的范式将指导下一代STEM MOOC的发展。拟议的全面、为期三年的Prime项目将在通过普渡X和edX合作伙伴关系提供的15个高级STEM(纳米技术相关)MOOC以及NanHUB-U和波音公司提供的15个高级STEM(纳米技术相关)MOOC的56,000多名学员中开发和测试情景评估框架。该项目将创建数据收集工具和评估方法，使各种形式的学习者数据具有意义，用于面向行动的决策，以支持在提供STEM MOOC方面的学习和机构目标。这项研究将采用多阶段混合方法来解决以下研究问题：(1)什么结构对学习者有贡献？高级STEM MOOC的行为和性能？(2)需要哪些信息来为利益相关者提供有关提供高级STEM MOOC的价值的决策？利益相关者的需求如何推动MOOC的设计？(3)什么是针对高级STEM MOOC的上下文相关的、可概括的评估框架？这项研究的结果可能会对评估领域做出重大的理论贡献，因为它开发了一个新的评估框架，将更有效地确定提供STEM MOOC的价值及其对学习结果的影响。

The Promoting Research and Innovation in Methodologies for Evaluation (PRIME) program supports research on evaluation with special emphasis on: (1) exploring innovative approaches for determining the impacts and usefulness of STEM education projects and programs; (2) building on and expanding the theoretical foundations for evaluating STEM education and workforce development initiatives, including translating and adapting approaches from other fields; and (3) growing the capacity and infrastructure of the evaluation field. The PI and co-PIs at Purdue University will undertake an important research study designed to create a new paradigm for evaluating STEM MOOCs that go beyond analyzing clickstream data and could be used for evidence-based decision making. The goal of this new paradigm is to contextualize evaluation metrics by taking into consideration learner characteristics, course characteristics, and stakeholder goals. It is envisioned that this new paradigm would guide the development of the next generation of STEM MOOCs.The proposed full-scale, three-year PRIME project will develop and test a contextualized evaluation framework on over 56,000 learners enrolled in 15 advanced STEM (nanotechnology-related) MOOCs offered through the Purdue X and edX partnership, in conjunction with nanoHUB-U and Boeing. The project will result in the creation of data collection instruments and an evaluation method that make sense of diverse forms of learner data for action-oriented decisions to support learning and institutional goals in providing STEM MOOCs. The research study will employ a multiphase mixed methods approach to address the following research questions: (1) What constructs contribute to learners? behavior and performance in advanced STEM MOOCs? (2) What information is needed to inform stakeholders decision-making regarding the value of offering advanced STEM MOOCs? How do stakeholders needs drive the design of MOOCs? (3) What is a context sensitive, generalizable framework of evaluation for advanced STEM MOOCs? The results from the research study have the potential to make significant theoretical contributions to the evaluation field by developing a novel evaluation framework that would be more effective for determining the value of offering STEM MOOCs and the impact on learning outcomes.