Improving STEM Program Quality in Out-of-School-Time: Tool Development and Validation.
Improving STEM Program Quality in Out-of-School-Time: Tool Development and Validation.
复制标题
提高课外时间的 STEM 项目质量：工具开发和验证。
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
G. Noam
中科院分区：
文献类型：
--
作者：
A. Shah;C. Wylie;D. Gitomer;G. Noam
在和校外时间（OST）的经验被视为补充，有助于学生的兴趣，参与和表现在科学，技术，工程和数学（STEM）。虽然现有的工具来衡量质量一般课后设置和其他人来衡量结构化的科学课堂经验，有必要为可靠的措施STEM计划的质量在OSTsettingssuchasafterschoolprograms，夏令营，博物馆或科学中心编程。在本文中，我们提出了成功的维度（DoS）工具，它定义了12个关键组成部分的非正式的，探索性的STEM编程，超越了学校的一天。此外，我们还根据两项研究（研究1（n = 284个观察结果）和研究2（n = 56个观察结果））提供了DoStool的有效性评估，包括可靠性证据。我们的研究结果表明，结构和有效性证据的一致性，以及为DoS提供的培训和认证程序，使其成为了解学生在学校以外的STEM体验质量的重要工具。像DoS这样的工具有几个方面的影响，包括能够跨项目进行国家比较，创建聚合数据库，提高项目质量和专业发展，以及将项目质量与学生水平的成果联系起来。在两个观察员进行两次观察后，使用G系数估计尺寸可靠性（见附录）。然后，我们能够进行可靠性（D）研究，以估计需要多少观测才能对模块的质量进行更可靠的估计（保持两个观测者不变）。这些分析（附录）表明，需要进行多次观察才能获得稳定的质量衡量标准，最稳定的衡量标准是使用探索性因素分析确定的双因素结构，为学习环境的质量和STEM意义创造的质量创建两个综合得分。有趣的是，需要更多的观察来理解STEM意义制造因素与学习环境因素：虽然四个观察可能会导致对学习环境因素的可靠估计，但考虑到当前研究中观察者之间的一致性水平，即使是10个观察也不足以用于STEM内容因素。鉴于这些发现，我们对培训，认证和校准过程进行了更改，并跟进研究2，以检查对评估员可靠性的影响。
In and out-of-school time (OST) experiences are viewed as complementary in contributing to students’ interest, engagement, and performance in science, technology, engineering, and mathematics (STEM). While tools exist to measure quality in general afterschool settings and others to measure structured science classroom experiences, there is a need for reliable measures of STEM program qual-ityinOSTsettingssuchasafterschoolprograms,summercamps,and museum or science center programming. In this paper we present the development of the Dimensions of Success (DoS) tool, which defines twelve key components of informal, exploratory STEM programming that goes beyond the school day. Additionally, we present avalidityargumentthatincludesreliabilityevidencefortheDoStool based on two studies: Study 1 ( n = 284 observations) and Study 2 ( n = 56 observations). Our findings suggest that the coherence of the constructs and validity evidence, as well as the training and certification procedures in place for DoS, make it an important tool to understand the quality of STEM experiences for youth beyond the school day. A tool like DoS has several implications, including the ability to make national comparisons across programs, create aggre-gate databases, improve program quality and professional development, as well as to link program quality to student-level outcomes. G-coefficient was used to estimate dimension reliabilities after two observations by two observers (see Appendix). We were then able to conduct a dependability (D) study to estimate how many observations would be needed to get a more reliable estimate of the quality of a module (keeping two observers constant). These analyses (Appendix) indicated that multiple observations are needed to get a stable measure of quality and that the most stable measure was to use the two-factor structure that was identified from the Exploratory Factor Analysis to create two composite scores for the quality of the learning environment and the quality of STEM meaning-making. Interestingly, more observations were needed to understand the STEM meaning-making factor versus the learning environment factors: While four observations would likely result in a reliable estimate of the learning environment factor, even 10 observations would be insufficient for the STEM content factor, given the levels of inter-observer agreement in the current study. Given these findings, we made changes to the training, certification, and calibration process and followed up with Study 2 to examine the impact on resulting assessor reliability.